1The upload process converts count files into reports, and
2uploads reports. There will be only one report, named YYYY-MM-DD.json,
3for a given day.
4
5First phase. Look at the localdir (os.UserConfigdir()/go/telemetry/local)
6and find all .count and .json files. Find the count files that are no
7longer active by looking at their metadata.
8
9Second phase. Group the inactive count files by their expiry date, and
10for each date generate the local report and the upload report. (The upload
11report only contains the counters in the upload configuration.) The upload
12report is saved in the local directory with a name like YYYY-MM-DD.json, if
13there is no file already existing with that name.
14If the local report is different, it is saved in the local directory
15with a name like local.YYYY-MM-DD.json. The new upload report is
16added to the list of .json files from the first phase. At this point
17the count files are no longer needed and can be deleted.
18
19Third phase. Look at the .json files in the list from the first phase.
20If the name starts with local, skip it. If there is a file with the
21identical name in the upload directory, remove the one in the local directory.
22Otherwise try to upload the one in the local directory,
23If the upload succeeds, move the file to the uploaded directory.
24
25
26There are various error conditions.
271. Several processes could look at localdir and see work to do.
281A. They could see different sets of expired count files for some day.
29 This could happen if another process is removing count files. In this
30 case there is already a YYYY-MM-DD.json file either in localdir
31 or updatedir, so the process seeing fewer count files will not generate
32 a report.
331B. They could see the same count files, and no report in either directory.
34 They will both generate (in memory) reports and check to see if there
35 is a YYYY-MM-DD.json file in either directory. They could both then
36 write two files with the same name, but different X values, but
37 otherwise the same contents. The X values are very close to the front
38 of the file. Assuming reasonable file system semantics one version of
39 the file will be written. To minimize this, just before writing reports
40 the code checks again to see if they exist.
411C. Once there is an existing well-formed file YYYY-MM-DD.json in localdir
42 eventually the upload will succeed, and the file will be moved to updatedir.
43 It is possible that other processes will not see the file in updatedir and
44 upload it again and also move it to uploaddir. This is harmless as all
45 the uploaded files are identical.
View as plain text