This post describes vixie-cron, not systemd-timers.
https://www.monkeyuser.com/2018/going-global/
Monkey User is always entertaining.
I wonder how on earth do you deal with a 30 or 45 minute offset in real life
Now this article makes sense to me; I was wondering what made 02:00 and 03:00 special, since the DST changes would be from 00:00 to 01:00 and from 00:00 to 23:00, as I'm used to since childhood here in Brazil. Perhaps some other countries change DST from 02:00 to 03:00 and vice versa?
My guess is that in the US they do the same but shifted by one.
And tbh, don't run jobs on the hour anyway. Everything kicks off then. Set stuff to run at 01:45 or 02:15 or something instead.
(None of these suggestions are time change related)
this is the way
This combines very simple configuration, while being predictable and spreading out timers well.
sleep $(( $(od -N1 -tuC -An /dev/urandom) % 60 ))m ;
TAI4LYFE!
I didn't use UTC because subtracting eight when reading raw logs is a lot harder than subtracting one.
They use UTC now, which makes sense since they have a global workforce and log reading interfaces that can automatically change timezones on display.
https://unix.stackexchange.com/questions/274816/how-to-conve...
In my old gig, jobs ran for many many HOURS and consumed most of the IO on the server (processing reams of data), which meant that when you got them running 2x because of overlap, it was a trainwreck.
That said, I absolutely prefer UTC times in logfiles. I'd rather do some math every time than make wrong assumptions about the local timezone of the file even once. (Even correctly indicated local times are more annoying, since I have the math for the offset of my timezone to UTC memorized, but not that of every possible server time to mine.)
How hard can it be to write a log cat utility in awk?
That sounds like a job for a human (at least the review part), not only a computer.
Case in point, before reddit I was at eBay, and we kept all those servers in Pacific time, since the entire technical workforce was in Pacific time, as well as all of the servers.
Making blanket statements like that without considering the context of the time is usually not a good idea. ;)
I was there back then, working for shops people have heard of, and I honestly don't know where you're getting this idea from. Some places did things wild and wacky when they were wee small, but most of us quickly learned that such shenanigans (like fun server naming conventions) start to fall apart and maybe we should do things differently.
Logs with weird dates on high demand production servers... less important.
Also important here is that the date stays the same! Even though I've gotten used to instinctively decoding UTC, that part is still frustrating sometimes.
Please stick with utc across the board people, someone someday may have to clean up your mess.
UTC was standardized in 1963 [0]
it was already a 40 year old standard at the time you're talking about.
awareness of UTC being the correct choice has definitely increased over time, but UTC being the correct choice has not changed.
you say reddit servers use UTC now, which implies there was a cutover at some point. were you still at reddit when that happened? were you still hands-on with server maintenance? any anecdotes or war stories from that switchover you want to share?
because I can easily imagine parts of the system taking a subtle dependency on Arizona being Reddit Standard Time, and the transition to UTC causing headaches when that assumption was broken. your memory of this "clever" trick might be different if you had to clean up the eventual mess as well.
Whenever you are unsure whether to use a clever solution or follow the globally accepted standard in your work as a DevOps or Software engineer, always choose the standard.
So that first alert that came in ?? minutes ago you need to align with the telemetry and logs in order to see what the servers were doing right before everything went to shit.
One is not necessarily and in all instances less of a mess to clean up behind than the other.
We had application logs and system level logs with different timestamp and someone decided a certain db column has to be a string of the format 'yyyy-mm-dd hh:mm:ss". You can imagine the confusion and the "fun" time we had while debugging logs from multiple systems at specific times.
Finally, one lead put their feet down and mandated that setting timezone to PST should be the first thing all applications do and db columns should be considered PST unless otherwise required.
That seems to me like a really bad mandate. If you are going to mandate something, mandate UTC. If someone forced me to change a system from UTC to something else, I’d not be very happy. It’s the kind of decision which makes you seriously question if you want to work there any more.
If nothing else, selecting UTC should be a very strong hint to any UI that I am capable of parsing YYYY-MM-DD and 24 hour times.
In firefox you can try it out in about:preferences by setting en-CA as the topmost option in "Choose your preferred language for displaying pages"
That's unfortunate, but then wouldn't the sane way for localization-aware software be to ask the user and not make any such assumptions?
> Assuming you're using the en-US locale, have you tried using the en-CA locale? It has ISO8601 formatted dates, and is otherwise pretty similar to the en-US locale.
Thanks, I'll give that a try tomorrow! If my log exploring UI of, well, not quite choice actually respects that, I'll be very grateful :)
I asked the HTML spec editors to fix this, but thus far they haven’t: https://github.com/whatwg/html/issues/6698
A plus would be showing both.
0 2 * * * run-parts /etc/periodic/daily
0 3 * * 6 run-parts /etc/periodic/weekly
https://www.google.com/search?q=busybox+cron+dst
"BusyBox’s crond is a lightweight implementation designed for embedded systems, so it doesn’t have all the timezone/DST logic of full vixie-cron or cronie. It just reads /etc/TZ or the system timezone at startup and then wakes up every minute to check if something should run."
just use UTC
anacron won't have the OP's 2am/3am issue because it locks jobs while they run, so multiple job spawns should be no-ops. Still seems like it would be good practice to, e.g. use UTC for the host system and do whatever else might help avoid the unnecessary re-spawns.
But the other thing that has changed since 2013 is that most of us are using autoscaling. There were times of day when there is supplemental server capacity to run periodic processes, and it made sense to schedule them during quiet daylight hours if they existed, but now the availability of bus numbers if and when a problem happens are more pressing. And they all outweigh concerns about DST. Instead of seeking the low traffic times of day you mainly need to avoid peak traffic.
The batch processing system I renovated to use 7% if the CPU hours of the original design, used the same fixed parallelism trick that crawlers use so as not to knock over your sites. They keep k requests in flight at all times and if your server responds faster, they issue more requests per second. If the production services had excess capacity the job would run in ten minutes. If response times are slow it takes 15. You’re inverting Little’s Law and keeping the amount of servers fixed and varying the duration.
It’s a pity that spooling up more servers to handle that job took a substantial fraction of that ten minutes or I could have gotten it to five (originally it was over 30 minutes and climbing toward 35). I tried at one point to move it entirely to a Bamboo agent but we underprovisioned them. If I’d had an agent with 2x the CPUs I could have shut down a service. I was on my way out the door before I realized that upgrading the bamboo agents would have cost us less than running that service and everyone’s builds would have been faster as a consequence. And probably let someone else do the same. False economies.
Is this what people actually want, though?
I'm actually working on a scheduled process right now deployed on a cloud scheduler that does not adjust for daylight saving time (it literally says "does not take into consideration daylight saving time" next to the time input).
Everybody wants the job to run four hours before the work day starts, every day of the year. I will have to change it on Sunday to ensure the job completes at the time the end users expect it. And again in March.
Am I missing something?
noir_lord•2h ago
Company went bust before it got fixed.
Just use UTC folks unless you have a really good idea why you shouldn't.
lxgr•2h ago
People generally expect things like usage limits to reset overnight, not at UTC midnight, and these are often implicitly tied to (the result of) batch jobs.
Financial jobs, including billing, is another big category.
latchkey•2h ago
The reporting job should be run from a server on UTC and on UTC time. The report itself can convert to whatever is local time.
gruez•2h ago
latchkey•2h ago
lxgr•2h ago
All of yours might well be, but you can't generalize that assumption to every domain and scenario.
ipaddr•2h ago
kjehjrktlhl•2h ago
lxgr•1h ago
gruez•1h ago
How are you going to do this? Clearly cron (with timezone set to UTC) doesn't work for this so you'll need another system. In that case why not have that multi-time zone scheduling system run your script?
latchkey•2h ago
ranger_danger•2h ago
But also, if the user can change the timezone themselves, now you have a condition where the user could keep their quota at bay for a very long time (if not forever) if they keep changing their timezone accordingly.
throwaway150•1h ago
How hard is to convert the UTC time to user's local time and perform quota reset?
I mean you should be doing this anyway. Even if you store everything in PST or EST or CST, you never know when you'll get customers from another part of the world and then you'll have to do this anyway. So why not do this already?
lxgr•1h ago
Probably roughly as hard as writing a timezone-aware scheduler that considers all edge cases around daylight savings time, i.e. probably possible but I'd try fairly hard to see if I can get around it.
One way of getting around it is to run your batch jobs in your "primary business timezone". Sure, you might outgrow that concept eventually, but many companies never do, and in that sense "running on UTC" seems similarly aspirational in spirit (albeit at a much smaller scale) to starting with high scalability.
gruez•1h ago
1. allowing users to change their timezone and tying it to when their quota reset sounds like a great way to add more edge cases and complexity. For instance, does messing with your time zone mean you can get your quota reset multiple times a day?
2. Not everyone operates some sort of global b2c SaaS that's timezone agnostic. For many enterprise backoffice tasks "6am HQ time" is far more reasonable than "6am/7am, dunno depends on whether daylight saving time is on or not". In this case having a server set to the HQ's timezone makes far more sense than doing UTC and trying to work around daylight saving issues.
freedomben•2h ago
Better to localize times for users on the client-side based on their local settings, IMHO
For quotas and usage limit resets it can be a little harder, but if it's a cron job anyway, my initial approach would be to just schedule it for a time that is close to midnight for the area your customers are in.
Kwpolska•2h ago
lxgr•2h ago
That sounds a bit like reimplementing a scheduler inside the scheduled task (which is ok if you don't trust your scheduler, but I'd avoid it if at all possible).
bell-cot•2h ago
- Check if run today, exiting on "yes"
- Run
- Update the last-run-on date
latchkey•1h ago
bell-cot•1h ago
But as soon as one job needs another to complete first, or two jobs shouldn't be run at the same time - things start getting messy.
lxgr•1h ago
I can think of several ways to mess that seemingly trivial calculation up just from the top of my head. (Consider changing timezones, increasing the interval the job is running to within half a day of scheduling period etc.)
skissane•28m ago
This doesn’t work as soon as you become a global business… because if you tell a customer in Asia that it resets at midnight in some US timezone, what are they going to think?
Many people will accept UTC because it is the international standard. Everyone will accept “let each customer pick the timezone that works best for them”. Saying “I’m going to use a US timezone because that is easiest for our US customers” risks sending a message to your (now or future) global customers that you view them as second-class citizens.
lxgr•17m ago
> Many people will accept UTC because it is the international standard.
I really wish that were true universally...
And I do think that UTC is actually somewhat Eurocentric! From personal experience, it's significantly easier to mentally add/subtract one or two hours (modulo 24) than 7 or more.
> Saying “I’m going to use a US timezone because that is easiest for our US customers” risks sending a message to your (now or future) global customers that you view them as second-class citizens.
Definitely, but on the other hand, I do think that picking your head office's time zone as the "canonical" one for resetting quotas etc. has some merit as well, if only because it makes the concepts of "today" and "tomorrow" work a bit better. (UTC midnight might be very unintuitive to both you and your customers.)
skissane•4m ago
What happens when you move your head office to a different time zone? e.g. Oracle moved their HQ from California to Texas (and have announced a further move to Tennessee, although last I heard that still hasn’t happened)
What happens when an M&A happens and you are now part of a much larger enterprise with a different HQ timezone, and systems with wildly inconsistent configured timezones due to having acquired multiple companies which had HQs in different time zones and all of which configured everything to use their HQ time?
> UTC midnight might be very unintuitive to both you and your customers
I live in eastern Australia, UTC+11 at the moment (UTC+10 during Southern Hemisphere winter). Resetting stuff at UTC midnight means at 11am my time, resetting stuff at US Eastern midnight means at 3pm my time - roughly equal in potential inconvenience, but I’ll be much more forgiving of the first than the second.
> And I do think that UTC is actually somewhat Eurocentric! From personal experience, it's significantly easier to mentally add/subtract one or two hours (modulo 24) than 7 or more.
For me, add/subtract 10 is pretty easy… and then just one more during DST. Does that make UTC “Australia-centric”? Or Guam-centric or Vladivostok-centric? (both of which are also UTC+10)
latchkey•2h ago
Just found out some guy decided to try to restart the company. Good luck. https://x.com/Austen/status/1981904435539280324
ta1243•2h ago
I get a daily status report of various things from our 24 hour operational management team which runs at 8am UK time every day. That means last week it ran at 0700UTC and this week at 0800UTC
This is built around operational events, shift changes, etc.
I've got another system which is in operation in Sydney from 0630 to 1630 local time, this means that maintence windows which overlap with UK shift patterns depend on the week but mean the system is operating 2130-0730 UK time at some times, 2030-0630 UK at others, and 1930-0530 at others.
UTC is not "the answer". Sometimes you want things running at a UTC time, sometimes you want them running at local time.
I have a regular meeting at 10am London time on Tuesday and Thursday. That can't be stored in UTC as it varies depending on the time of the year. It has to have the timezone stored and actioned.
latchkey•2h ago
But if the job is set to run in UK time and the system clock jumps around because of TZ changes, it will run twice or not at all.
Having a stable zero based time that doesn't move around by external forces that you can't control is "the answer."
ta1243•2h ago
How about I use some form of library to do it. I tell it I want to run at 0800 London time, and it runs at 0800 London time
If I tell it I want to run at 0130 London time (or 0330 Athens time) I still have a problem -- do I run it twice when the clocks go back, do I skip it when clocks go forward?
But that's a business logic problem, and defining it as UTC and having another job to update the time twice a year doesn't actually solve the question of "what do I do at this point".
latchkey•2h ago
ipaddr•2h ago
baq•2h ago
> If I want a report of what happened at a specific time I need that report at local time
this is hard when a particular hour can happen twice in a day right?
jedberg•2h ago
If my whole team is based in Pacific Time, I'm gonna use something close to PT so I don't have to do hard math when reading raw logs.
If we're all US based, I still want something in the USA. The math is easier.
kjehjrktlhl•2h ago
jrockway•2h ago
jedberg•2h ago
throwaway150•1h ago
Nobody hurt them. Please don't do this kind of childish rebuttals. It looks really silly on HN. If you don't want to use UTC, that's one thing. You do you. But there are reasons for sane professionals (who weren't hurt by anybody) to expect their coworkers to use UTC when possible. Because using local timezone leads to some problems including the one in TFA.
hobs•1h ago
Just because it worked once doesn't mean it's good.
anonymars•45m ago
UTC is the right choice for representing specific moments in time (so yes, log timestamps) but there are so many pitfalls outside of that narrow use case
onraglanroad•2h ago
BenjiWiebe•1h ago
If it's local time I know instantly when something happened, without having to do mental math.
Is there anything wrong with ISO8601 (including timezone offset) for storing times? They're in my local time, but they can be accurately converted to any other timezone.
onraglanroad•1h ago
It's just easier for everyone if we agree on UTC and everyone knows their own offset.
jedberg•1h ago
onraglanroad•42m ago
I'd expect everyone who works in computer related jobs to know their UTC offset.
adammarples•1h ago
jedberg•1h ago
yabones•2h ago
derwiki•1h ago
schraitle•1h ago
Right now it reads "EDT, UTC-4, until: Sun, Nov 2"
Going to add a clock next to it now, that's a great idea
ZeWaka•1h ago
gtowey•1h ago
walthamstow•10m ago
lozf•10m ago
TZ="Etc/UTC" xclock -analog -update 1 -norender -hl grey -fg grey -bg black
(or similar) would also suffice.
nodesocket•2h ago
SecondHandTofu•2h ago
jacobsenscott•1h ago