Hey I am trying to build an HTTP client, and I can't seem to figure out why I am not getting the code "304, not modified". The last modified date for the file is in 2007 and I am accessing it 10 yrs later.
Here is the output
HTTP/1.1 200 OK
Date: Tue, 03 Oct 2017 21:50:33 GMT
Server: Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.2k-fips
Last-Modified: Fri, 31 Aug 2007 04:21:06 GMT
ETag: "c12-438f726ceb080"
Accept-Ranges: bytes
Content-Length: 3090
Content-Type: image/gif
and the java code for reference, I am using sockets.
Date d = new Date();
outputStream.print("HEAD "+ "/" + pathName + " HTTP/1.1\r\n");
outputStream.print("If-Modified-Since: " + d.toString() + "\r\n");
outputStream.print("Host: " + hostString+"\r\n");
outputStream.print("\r\n");
outputStream.flush();
any help would be appreciated, I am new to HTTP clients.
You might need to format the date you're using for the "If-Modified-Since" header. Here's the syntax (taken from here):
If-Modified-Since: day-name, day month year hour:minute:second GMT
Example:
If-Modified-Since: Wed, 21 Oct 2015 07:28:00 GMT
Related
According to https://developers.google.com/gmail/api/guides/filtering, if you search for a date, the PST timezone is used.
I have an email with:
Date: Tue, 27 Jul 2021 06:07:49 GMT (06:07 AM)
If you convert this to the PST timezone, it should be:
Date: Mon, 26 Jul 2021 23:07:49 PST (11:07 PM)
Now when i use the query
after:2021/07/27
the email is found but shouldn't be according to the definition (also when i search directly on gmail.com).
For the query
before:2021/07/27
nothing is returned.
I didn't find a description yet on how to correctly search by a date and which timezones are really applied.
I'm using google-api-services-gmail-v1-rev110-1.25.0.jar.
Additional info:
String account = "myemail#gmail.com";
List<String> labelIds = new ArrayList<>();
labelIds.add("Label_mylabelid");
String query = "after:2021/07/27";
List<Message> timeZoneMsgs = gmail.users().messages().list(account).setQ(query).setLabelIds(labelIds).execute().getMessages();
I ran another test: I scheduled some emails to be sent between 11:30pm and 01:00am on the next day.
Here are the date headers (directly out of the payload) of the messages:
Scheduled mail 1
Date: Fri, 24 Dec 2021 23:30:00 +0100
Scheduled mail 2
Date: Fri, 24 Dec 2021 23:45:00 +0100
Scheduled mail 3
Date: Sat, 25 Dec 2021 00:00:00 +0100
Scheduled mail 4
Date: Sat, 25 Dec 2021 00:15:00 +0100
Scheduled mail 5
Date: Sat, 25 Dec 2021 00:30:00 +0100
Scheduled mail 6
Date: Sat, 25 Dec 2021 00:45:00 +0100
Scheduled mail 7
Date: Sat, 25 Dec 2021 01:00:00 +0100
If I search with
after:2021/12/25
directly on gmail.com - i get the messages 3 to 7 and if i search with
before:2021/12/25
I get 1 and 2, which is both correct.
BUT when I do the same with the java call, I only get message 7 (1am) for "after:2021/12/25" and messages 1-6 for "before:2021/12/25".
Another example:
I also tried searching with the epoch time in UTC:
Calendar calendar = Calendar.getInstance(TimeZone.getTimeZone("UTC"));
calendar.clear();
calendar.set(2021, Calendar.DECEMBER, 25);
long secondsSinceEpoch = calendar.getTimeInMillis() / 1000L; // 1640390400
Here, I get the same result for direct gmail.com search and java search: Message 7 (1pm) for "after:1640390400" and 1-6 for "before:1640390400".
If I try this for PST (1640419200), I get no messages for "after:1640419200" and all of them for "before:1640419200".
As the documentation states:
To specify accurate dates for other timezones pass the value in seconds instead.
This is a Java sample code, is tested and working .For the example I get all the messages from yesterday 23:00 to today 13:00. Tip: You can use me as a keyword to your email account:
long startTimeEpoch = LocalDateTime.parse("2021-12-27T23:00",
DateTimeFormatter.ofPattern("yyyy-MM-dd'T'HH:mm")).toEpochSecond(ZoneOffset.UTC);
long endTimeEpoch = LocalDateTime.parse("2021-12-28T13:00",
DateTimeFormatter.ofPattern("yyyy-MM-dd'T'HH:mm")).toEpochSecond(ZoneOffset.UTC);
String query = String.format("after:%s before:%s", startTimeEpoch, endTimeEpoch);
String account = "me";
System.out.println(query);
List < String > labelIds = new ArrayList < > ();
labelIds.add("Label_mylabelid");
List < Message > timeZoneMsgs = service.users().messages().list(account)
.setQ(query)
.execute()
.getMessages();
System.out.println(timeZoneMsgs);
Documentation:
Gmail API Javadoc documentation
LocalDateTime
DateTimeFormatter
I'm downloading a JAR file, and would like to utilize If-Modified-Since so I don't get the whole file if I don't need it, but for some reason my vanilla Apache (afaik) isn't returning the 304 correctly.
This is from wireshark:
GET /whatever.jar HTTP/1.1
If-Modified-Since: Sat, 04 Jan 2014 21:46:26 GMT
User-Agent: Jakarta Commons-HttpClient/3.1
Host: example.com
HTTP/1.1 200 OK
Date: Sat, 04 Jan 2014 20:32:31 GMT
Server: Apache/2.2.4 (Unix) mod_ssl/2.2.4 OpenSSL/0.9.8e DAV/2 mod_jk/1.2.26 PHP/5.3.6 SVN/1.4.4
Last-Modified: Sat, 04 Jan 2014 19:13:14 GMT
ETag: "b6c037-1ddad9f-d17a6680"
Accept-Ranges: bytes
Content-Length: 31305119
Vary: User-Agent
Content-Type: text/plain
... [bunch of bytes] ...
There aren't other headers I need to specify, is there? Am I missing a module that Apache needs in order to read this header correctly?
Any other thoughts or suggestions?
Here is my Java code, for reference:
File jarFile = new File(filePath);
GetMethod get = new GetMethod(downloadUrl);
Date lastModified = new Date(jarFile.lastModified());
get.setRequestHeader("If-Modified-Since", DateUtil.formatDate(lastModified));
HttpClient client = new HttpClient();
int code = client.executeMethod(get);
UPDATE: Solution
The If-Modified-Date needed to exactly match the server, and I achieved this by explicitly setting the lastModifiedDate on the downloaded file:
String serverModified = get.getResponseHeader("Last-Modified").getValue();
jarFile.setLastModified(DateUtil.parseDate(serverModified).getTime());
After doing this, subsequent calls would not download the file.
In order to use the "If-Modified-Since" header, you must send an identical header value as the "Last-Modified" header, that is Sat, 04 Jan 2014 19:13:14 GMT != Sat, 04 Jan 2014 21:46:26 GMT. Apache cannot guarantee the file wasn't modified and given a past time on purpose (perhaps through a version control roll-back).
If you want, you may check the "Last-Modified" header on the client side, by using a HeadMethod first to avoid "getting" the resource if it hasn't been modified. Then you would use a "GetMethod" if it has been modified.
See RFC2616 - Section 9, "HTTP/1.1: Method Definitions" for more.
I am trying to serve some assets using a Spring MVC controller. My assets are database managed and thus have to be served this way. The service looks up the metadata of the asset from the database, reads the file from file system and builds the response.
Here is how my controller looks like.
#Controller
#RequestMapping("/assets")
public class AssetController {
#Autowired
private AssetService assetService;
#RequestMapping("/{assetName:.+}")
public ResponseEntity<byte[]> getAsset(#PathVariable("assetName") String assetName) throws FileNotFoundException, IOException {
Asset asset = assetService.findByName(assetName);
HttpHeaders headers = new HttpHeaders();
headers.setContentType(MediaType.valueOf(asset.getContentType()));
headers.setCacheControl("max-age=1209600");
headers.setLastModified(asset.getModifiedOn().getTime()); // always in the past
return new ResponseEntity<byte[]>(assetService.toBytes(asset), headers, OK);
}
}
Seems simple and straightforward enough? One would hope to see the browser caching the images. But despite trying all combinations of Cache-Control, Expires, Last-Modified-On and ETag, I have had no success.
Below are the HTTP headers (irrelevant headers removed) spit out during two successive requests.
GET /adarshr-web/assets/Acer.png HTTP/1.1
Host: localhost:8080
Pragma: no-cache
Cache-Control: no-cache
HTTP/1.1 200 OK
Cache-Control: max-age=1209600
Last-Modified: Sun, 21 Jul 2013 11:56:32 GMT
Content-Type: image/png
Date: Tue, 23 Jul 2013 21:22:58 GMT
----------------------------------------------------------
GET /adarshr-web/assets/Acer.png HTTP/1.1
Host: localhost:8080
If-Modified-Since: Sun, 21 Jul 2013 11:56:32 GMT
Cache-Control: max-age=0
HTTP/1.1 200 OK <-- Why not 304 Not Modified?
Cache-Control: max-age=1209600
Last-Modified: Sun, 21 Jul 2013 11:56:32 GMT
Content-Type: image/png
Date: Tue, 23 Jul 2013 21:23:03 GMT
However, when I try the same sequence (Ctrl + F5 for first request and F5 for subsequent ones) on URLs such as
http://www.google.co.uk/images/srpr/logo4w.png (Google's logo)
http://fbstatic-a.akamaihd.net/rsrc.php/v2/yI/r/0PsXdTWc41M.png (Facebook's mobile image)
I see the headers such as these (shown for the Facebook URL) which indicate that the response is being cached by the browser.
GET /rsrc.php/v2/yI/r/0PsXdTWc41M.png HTTP/1.1
Host: fbstatic-a.akamaihd.net
Pragma: no-cache
Cache-Control: no-cache
HTTP/1.1 200 OK
Content-Type: image/png
Last-Modified: Sat, 15 Jun 2013 00:48:42 GMT
Cache-Control: public, max-age=31535893
Expires: Wed, 23 Jul 2014 21:27:47 GMT
Date: Tue, 23 Jul 2013 21:29:34 GMT
----------------------------------------------------------
GET /rsrc.php/v2/yI/r/0PsXdTWc41M.png HTTP/1.1
Host: fbstatic-a.akamaihd.net
If-Modified-Since: Sat, 15 Jun 2013 00:48:42 GMT
Cache-Control: max-age=0
HTTP/1.1 304 Not Modified <-- Note this
Content-Type: image/png
Last-Modified: Sat, 15 Jun 2013 00:48:42 GMT
Cache-Control: public, max-age=31535892
Expires: Wed, 23 Jul 2014 21:27:47 GMT
Date: Tue, 23 Jul 2013 21:29:35 GMT
Notes:
I don't have an <mvc:resources /> section in my Spring config since I am doing exactly the same in my controller. Even adding it doesn't make any difference.
I don't have a org.springframework.web.servlet.mvc.WebContentInterceptor defined in the Spring config again for the reasons above. I have tried adding one with no gain.
I have tried all methods explained in https://developers.google.com/speed/docs/best-practices/caching.
I can replicate this across all browsers.
You'll have to implement the check of the last modified, fortunately Spring makes that pretty easy.
From the Spring Framework Reference
#RequestMapping
public String myHandleMethod(WebRequest webRequest, Model model) {
long lastModified = // 1. application-specific calculation
if (request.checkNotModified(lastModified)) {
// 2. shortcut exit - no further processing necessary
return null;
}
// 3. or otherwise further request processing, actually preparing content
model.addAttribute(...);
return "myViewName";
}
When creating the HTTP Response manually, how can one get Server and ETag
* HTTP/1.1 200 OK
* Date: Mon, 23 Apr 2012 23:44:52 GMT
* Server: Apache/2.2.3 (Red Hat) <-----
* Last-Modified: Fri, 16 Sep 2005 18:08:50 GMT
* ETag: "421142-2f-400e77c517080" <-----
* Accept-Ranges: bytes
* Content-Length: 47
* Content-Type: text/plain
* Connection: close
"Server" is whatever your HTTP server wants to name/identify itself. I.e. "Zumgto Surver 4.5".
"ETag" identifies "version" of particular item, so as long as your server can reasonable say "this ETag corresponds to current version" you can send pretty much anything. I.e. "v3345", or hash of the item... Totally optional if you don't support "If-None-Match" header in requests.
Neither is required. You can make up your own sever tag using the same format above. Omit the eTag or just generate your own. You could use the current timestamp or a constant. The following formats should work.
Server: Program/version (O/S)
ETag: "Timestamp"
Is there any available email parser library or implementation tutorial (without JavaMail API) to parse messages of the fallowing format to get their fields such as FROM / SUBJECT /etc ?
I want to parse a text file of these messages received via the POP3 protocol in my application and finally display them in a html file.
a sample message is :
-------------------- messsage 1, size=4234 --------------------
Return-path: <yyy#gmail.com>
Received: from mail-iy0-f171.google.com (mail-iy0-f171.google.com [209.85.210.171])
by xxxmail.com (xxxmail.com)
(MDaemon PRO v11.0.0)
with ESMTP id md50000000053.msg
for <xx#xxxmail.com>; Mon, 12 Dec 2011 22:55:39 +0330
Authentication-Results: xxxmail.com
spf=pass smtp.mail=haghighi.fa#gmail.com;
x-ip-ptr=pass dns.ptr=mail-iy0-f171.google.com (ip=x.x.x.x);
x-ip-helo=pass smtp.helo=mail-iy0-f171.google.com (ip=x.x.x.x);
x-ip-mail=hardfail smtp.mail=haghighi.fa#gmail.com (does not match x.x.x.x);
x-vbr=hardfail header.vbr-info=gmail.com (domain not recognized);
dkim=pass header.d=gmail.com (b=Cv+42gRZMW; 1:0:good);
Received-SPF: pass (xxxmail.com: domain of yyy#gmail.com
designates x.x.x.x as permitted sender)
x-spf-client=MDaemon.PRO.v11.0.0
receiver=xxxmail.com
client-ip=x.x.x.x
envelope-from=<yyy#gmail.com>
helo=mail-iy0-f171.google.com
X-Spam-Processed: xxxmail.com, Mon, 12 Dec 2011 22:55:39 +0330
(not processed: sender in recipient's private address book)
X-MDPtrLookup-Result: pass dns.ptr=mail-iy0-f171.google.com (ip=x.x.x.x) (xxxmail.com)
X-MDHeloLookup-Result: pass smtp.helo=mail-iy0-f171.google.com (ip=x.x.x.x) (xxxmail.com)
X-MDMailLookup-Result: hardfail smtp.mail=yyy#gmail.com (does not match x.x.x.x) (xxxmail.com)
X-MDDKIM-Result: unapproved (xxxmail.com)
X-MDVBR-Result: not certified (xxxmail.com)
X-MDSPF-Result: unapproved (xxxmail.com)
X-Rcpt-To: xxx#xxxmail.com
X-MDRcpt-To: xxx#xxxmail.com
X-MDRemoteIP: x.x.x.x
X-Return-Path: yyy#gmail.com
X-Envelope-From: yyya#gmail.com
X-MDaemon-Deliver-To: xxx#xxxmail.com
Received: by iaen33 with SMTP id n33so35124825iae.30
for <xxx#xxxmail.com>; Mon, 12 Dec 2011 11:28:52 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=gmail.com; s=gamma;
h=mime-version:in-reply-to:references:date:message-id:subject:from:to
:content-type;
bh=SZ4ok7NaD3MuwvPJVzJAtsCaqSMeAJJ6Kq3AFI1elK8=;
b=Cv+42gRZMW8gRSzCX11ToY/EnHTqzO2E5cWkJsnwj6JCapz5GXG2iIadjBkHyKuGkE
i5cjPQyZXhHBg9ZfKRaWViSMqiaySvak7WA+yVf65JB2zEHykysFHbqbfPLG2CaRxXHi
PVfJVURd5MHCLpSyCxeW25slIBOBqpWabuWj4=
MIME-Version: 1.0
Received: by 10.42.151.68 with SMTP id d4mr13014719icw.36.1323718132412; Mon,
12 Dec 2011 11:28:52 -0800 (PST)
Received: by 10.43.49.67 with HTTP; Mon, 12 Dec 2011 11:28:52 -0800 (PST)
In-Reply-To: <4ee65516.85310e0a.43bf.238fSMTPIN_ADDED#mx.google.com>
References: <4ee65516.85310e0a.43bf.238fSMTPIN_ADDED#mx.google.com>
Date: Mon, 12 Dec 2011 11:28:52 -0800
Message-ID: <CAPGX6DbXDV-wFmbMcaw1rQVvrZGbBOnqPpJSNkwuhbrBFa=TRQ#mail.gmail.com>
Subject: Re: Hi :D
From: y y <yyy#gmail.com>
To: xxx#xxxmail.com
Content-Type: multipart/alternative; boundary=90e6ba6e872e10b18504b3ea252a
X-Antivirus: avast! (VPS 111228-0, 12/28/2011), Inbound message
X-Antivirus-Status: Clean
--90e6ba6e872e10b18504b3ea252a
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
=D8=A7=D9=84=D9=84=D9=87 =D9=88=D8=A7=D9=82=D8=B9=D9=86 =D8=A7=DA=A9=D8=A8=
=D8=B1=D9=87 :=D8=AF=DB=8C
On Mon, Dec 12, 2011 at 12:00 AM, <xxx#xxxmail.com> wrote:
> Hi! this is the main text of the message :)
>
>
>
I've found these examples for both POP3 and SMTP:
http://inetjava.sourceforge.net/lectures/part1_sockets/InetJava-1.8-Email-Examples.html
They don't rely on Java Mail, they use just a plain socket. Look at getMessageHeaders() to obtain the information about the fields FROM and SUBJECT.