We are here to talk about the y2k38, the second installment of the end of time
for computers, with the first being the infamous y2k who was responsibly fixed
one chunk of code at a time by programmers all around the world, well before
the expected deadline.

Our new "2038 problem" lies in the fact that a lot of computers keep the time
internally as the number of seconds that has passed since 1970-01-01 00:00:00
UTC, storing that integer in a nice 32bit signed integer which gives support
for the immensely wide range of:
  * 1901-12-13 20:45:52 UTC to
  * 2038-01-19 03:14:07 UTC.

More than enough if you were to design this method for time keeping during
1969.  Yet with the year 2038 getting closer and it's clear that somehow we
must change how to store these dates.

There were several proposals:

  1. Let's use a 64bits integer instead!
     * this is one of the reasonable solutions, with the new epochalypse
       now becoming 292277026596-12-04 15:30:08 UTC.  However, it feels like
       we are wasting space by allowing a so far away date.

  2. Let's use a 64bits integer but instead let's save the number of
     milliseconds from the epoch!
     * And now we have a much closer 292278994-08-17 07:12:55.807 UTC, but
       still several times beyond the estimated date when the Sun will eat
       our planet.
     * Even when storing microseconds we still get a reasonable limit:
       294247-01-10 04:00:54 UTC
     * Or when storing the number of nanoseconds: 2262-04-11 23:47:16 UTC,
       which once again is more than enough if we consider the speed at which
       us humans are causing global warming.

Then it becomes obvious to consider using a standard 64bits ieee-754 floating
point number! that way we would not need to care about whether to save the
number of s, ms, µs or ns, we could just keep the number of seconds in the
integral part and use the remaining bits for the fraction of seconds.



How do ieee754 floating point numbers (from now on just floats) work?

They encode rational numbers using scientific notation in binary (base 2).
To do this we can use the following algorithm:

function convert_to_floating_point(n, fractional_bits)
  local buffer = {}
  exponent = 0
  -- Handle zero, nan and infinity, negative numbers:
  if n == 0 then return 'zero' end
  if type(n) ~= 'number' or n ~= n then return 'nan' end
  if n < 0 then table.insert(buffer, '-'); n=-n end
  if n == math.huge then return table.concat(buffer) .. 'inf' end

  -- Now let's move the comma to the left on numbers larger than the base (2):
  while n >= 2 do
    n = n/2
    exponent = exponent + 1
  end

  -- And now let's do the opposite for numbers smaller than 1:
  while n < 1 do
    n = n*2
    exponent = exponent - 1
  end

  -- Now our number is in the range 1 <= n < 2, and we have the exponent ready,
  -- so we know that it starts with '1.'
  table.insert(buffer, '1.')
  n = (n - 1) * 2

  -- so the next phase is taking fractional_bits out of it:
  for i = 1, fractional_bits do
    if n >= 1 then
      table.insert(buffer, '1')
      n = (n - 1) * 2
    else
      table.insert(buffer, '0')
      n = n * 2
    end
  end
  return table.concat(buffer) .. ' * 2^' .. exponent
end


And so, let's test convert 9.25: 1.001010000 * 2^3

Which makes sense, as 9.25 = 9 + 1/4; in binary: 1001 + 1/100 = 1001.01

Note that since scientific binary numbers always begin with "1." (except for
the zero), ieee-754 floats ignore this "1." when storing them as it becomes
redundant.  What they do contain is the following:

  * The sign bit (0 for positive, 1 for negative)
  * Exponent bits:
      We don't need to know how these bits are used, but in case you are
      curious:
      We first add a constant to the exponent called "exponent bias", this bias
      depends on the size of this field:
      on 32bit floats the exponent has only 8 bits and the bias is 127, while
      on 64bit floats the exponent has 11 bits with 1023 as the bias.
      Then we clamp its value to the maximum and minimum representable
      exponents: if it ends up at zero we shift the mantissa to the right by
      how much we went below zero to represent subnormal numbers, and if it
      went beyond the max value we store all ones in the mantissa as well to
      represent ±infinity.
  * Mantissa bits:
      As we have mentioned we only keep the fractional part, padding with
      zeros at the end, that's it.

Since in 64bit floats we have 52 bits to store the fractional part, this means
that any bits to the right of those 52 bits will be lost.

Moreover particular the precision of the floating point number will depend on
the size of the integer part!, that would be the number of seconds from epoch,
and so, these are the date ranges for which we would expect a given precision:

1970-01-01 00:00:02 .. 1970-01-01 00:00:03 -> 4.44089e-16
1970-01-01 00:00:04 .. 1970-01-01 00:00:07 -> 8.88178e-16
1970-01-01 00:00:08 .. 1970-01-01 00:00:15 -> 1.77636e-15
1970-01-01 00:00:16 .. 1970-01-01 00:00:31 -> 3.55271e-15
1970-01-01 00:00:32 .. 1970-01-01 00:01:03 -> 7.10543e-15
1970-01-01 00:01:04 .. 1970-01-01 00:02:07 -> 1.42109e-14
1970-01-01 00:02:08 .. 1970-01-01 00:04:15 -> 2.84217e-14
1970-01-01 00:04:16 .. 1970-01-01 00:08:31 -> 5.68434e-14
1970-01-01 00:08:32 .. 1970-01-01 00:17:03 -> 1.13687e-13
1970-01-01 00:17:04 .. 1970-01-01 00:34:07 -> 2.27374e-13
1970-01-01 00:34:08 .. 1970-01-01 01:08:15 -> 4.54747e-13
1970-01-01 01:08:16 .. 1970-01-01 02:16:31 -> 9.09495e-13
1970-01-01 02:16:32 .. 1970-01-01 04:33:03 -> 1.81899e-12
1970-01-01 04:33:04 .. 1970-01-01 09:06:07 -> 3.63798e-12
1970-01-01 09:06:08 .. 1970-01-01 18:12:15 -> 7.27596e-12
1970-01-01 18:12:16 .. 1970-01-02 12:24:31 -> 1.45519e-11
1970-01-02 12:24:32 .. 1970-01-04 00:49:03 -> 2.91038e-11
1970-01-04 00:49:04 .. 1970-01-07 01:38:07 -> 5.82077e-11
1970-01-07 01:38:08 .. 1970-01-13 03:16:15 -> 1.16415e-10
1970-01-13 03:16:16 .. 1970-01-25 06:32:31 -> 2.32831e-10
1970-01-25 06:32:32 .. 1970-02-18 13:05:03 -> 4.65661e-10
1970-02-18 13:05:04 .. 1970-04-08 02:10:07 -> 9.31323e-10
1970-04-08 02:10:08 .. 1970-07-14 04:20:15 -> 1.86265e-09
1970-07-14 04:20:16 .. 1971-01-24 08:40:31 -> 3.72529e-09
1971-01-24 08:40:32 .. 1972-02-16 17:21:03 -> 7.45058e-09
1972-02-16 17:21:04 .. 1974-04-03 10:42:07 -> 1.49012e-08
1974-04-03 10:42:08 .. 1978-07-04 21:24:15 -> 2.98023e-08
1978-07-04 21:24:16 .. 1987-01-05 18:48:31 -> 5.96046e-08
1987-01-05 18:48:32 .. 2004-01-10 13:37:03 -> 1.19209e-07
2004-01-10 13:37:04 .. 2038-01-19 03:14:07 -> 2.38419e-07
2038-01-19 03:14:08 .. 2106-02-07 06:28:15 -> 4.76837e-07
2106-02-07 06:28:16 .. 2242-03-16 12:56:31 -> 9.53674e-07
2242-03-16 12:56:32 .. 2514-05-30 01:53:03 -> 1.90735e-06
2514-05-30 01:53:04 .. 3058-10-26 03:46:07 -> 3.8147e-06
3058-10-26 03:46:08 .. 4147-08-20 07:32:15 -> 7.62939e-06
4147-08-20 07:32:16 .. 6325-04-08 15:04:31 -> 1.52588e-05
6325-04-08 15:04:32 .. 10680-07-14 06:09:03 -> 3.05176e-05
10680-07-14 06:09:04 .. 19391-01-25 12:18:07 -> 6.10352e-05
19391-01-25 12:18:08 .. 36812-02-20 00:36:15 -> 0.00012207
36812-02-20 00:36:16 .. 71654-04-10 01:12:31 -> 0.000244141
71654-04-10 01:12:32 .. 141338-07-19 02:25:03 -> 0.000488281
141338-07-19 02:25:04 .. 280707-02-04 04:50:07 -> 0.000976563
280707-02-04 04:50:08 .. 559444-03-08 09:40:15 -> 0.00195313
559444-03-08 09:40:16 .. 1116918-05-14 19:20:31 -> 0.00390625
1116918-05-14 19:20:32 .. 2231866-09-25 14:41:03 -> 0.0078125
2231866-09-25 14:41:04 .. 4461763-06-20 05:22:07 -> 0.015625
4461763-06-20 05:22:08 .. 8921556-12-07 10:44:15 -> 0.03125
8921556-12-07 10:44:16 .. 17841143-11-13 21:28:31 -> 0.0625
17841143-11-13 21:28:32 .. 35680317-09-25 18:57:03 -> 0.125
35680317-09-25 18:57:04 .. 71358665-06-19 13:54:07 ->  0.25
71358665-06-19 13:54:08 .. 142715360-12-06 03:48:15 ->   0.5
142715360-12-06 03:48:16 .. 285428751-11-12 07:36:31 ->   1

In other words, you will be/were able to have:
  nanoseconds precision until 1970-04-08
  microseconds precision until 2242-03-16
  milliseconds precision until 280707-02-04
  and seconds precision until 285428751-11-12



If you want to store the time in a 64bits floating point number, remember:

  * You can relatively safely use it to store microseconds with perfect
    precision until 2242-03-16, but it will start skipping microseconds
    from that date (probably not an issue right now).

  * It's totally safe to store times with milliseconds precision.
    I don't think that any of the hardware that we currently have or run
    will be able to reach the year 280707.

Postscript:

  If you want to know who uses this format, take a look at the following links:

  https://w3.impa.br/~diego/software/luasocket/socket.html#gettime
  https://docs.python.org/3/library/time.html#time.time
  https://docs.ruby-lang.org/en/master/Time.html#method-i-to_f
  https://www.postgresql.org/docs/current/functions-datetime.html#FUNCTIONS-DATETIME-EXTRACT
    * try: select pg_typeof(extract(epoch from now())); -- if you don't believe

Suggested material for reading on the topic:
  https://www.leebutterman.com/2021/02/01/store-your-unix-epoch-times-as-float64.html
  https://en.wikipedia.org/wiki/IEEE_754
  https://en.wikipedia.org/wiki/Time_formatting_and_storage_bugs
  https://en.wikipedia.org/wiki/Year_2038_problem
  https://ocw.mit.edu/courses/6-00-introduction-to-computer-science-and-programming-fall-2008/resources/lecture-5/
    * this is an introduction to floating point numbers in python.  I add this
      link because the other videos in the course are also interesting for
      anyone starting with programming, even though it's a bit dated by now.

<?

function title(text)
  return image(320, 24)
  :color(5):rect(0, 0, 320, 24)
  :color(7):text(8, 8, text):render()
end

return title 'OTHER 2038 ISSUES'

?>

We are here to talk about the y2k38, the second installment of the end of time
for computers, with the first being the infamous y2k who was responsibly fixed
one chunk of code at a time by programmers all around the world, well before
the expected deadline.

Our new "2038 problem" lies in the fact that a lot of computers keep the time
internally as the number of seconds that has passed since 1970-01-01 00:00:00
UTC, storing that integer in a nice 32bit signed integer which gives support
for the immensely wide range of:
  * 1901-12-13 20:45:52 UTC to
  * 2038-01-19 03:14:07 UTC.

More than enough if you were to design this method for time keeping during
1969.  Yet with the year 2038 getting closer and it's clear that somehow we
must change how to store these dates.

There were several proposals:

  1. Let's use a 64bits integer instead!
     * this is one of the reasonable solutions, with the new epochalypse
       now becoming 292277026596-12-04 15:30:08 UTC.  However, it feels like
       we are wasting space by allowing a so far away date.

  2. Let's use a 64bits integer but instead let's save the number of
     milliseconds from the epoch!
     * And now we have a much closer 292278994-08-17 07:12:55.807 UTC, but
       still several times beyond the estimated date when the Sun will eat
       our planet.
     * Even when storing microseconds we still get a reasonable limit:
       294247-01-10 04:00:54 UTC
     * Or when storing the number of nanoseconds: 2262-04-11 23:47:16 UTC,
       which once again is more than enough if we consider the speed at which
       us humans are causing global warming.

Then it becomes obvious to consider using a standard 64bits ieee-754 floating
point number! that way we would not need to care about whether to save the
number of s, ms, µs or ns, we could just keep the number of seconds in the
integral part and use the remaining bits for the fraction of seconds.

<?

    return title 'INTRODUCTION TO FLOATING POINT NUMBERS'

?>

How do ieee754 floating point numbers (from now on just floats) work?

They encode rational numbers using scientific notation in binary (base 2).
To do this we can use the following algorithm:

<? code = [[
function convert_to_floating_point(n, fractional_bits)
  local buffer = {}
  exponent = 0
  -- Handle zero, nan and infinity, negative numbers:
  if n == 0 then return 'zero' end
  if type(n) ~= 'number' or n ~= n then return 'nan' end
  if n < 0 then table.insert(buffer, '-'); n=-n end
  if n == math.huge then return table.concat(buffer) .. 'inf' end

  -- Now let's move the comma to the left on numbers larger than the base (2):
  while n >= 2 do
    n = n/2
    exponent = exponent + 1
  end

  -- And now let's do the opposite for numbers smaller than 1:
  while n < 1 do
    n = n*2
    exponent = exponent - 1
  end

  -- Now our number is in the range 1 <= n < 2, and we have the exponent ready,
  -- so we know that it starts with '1.'
  table.insert(buffer, '1.')
  n = (n - 1) * 2

  -- so the next phase is taking fractional_bits out of it:
  for i = 1, fractional_bits do
    if n >= 1 then
      table.insert(buffer, '1')
      n = (n - 1) * 2
    else
      table.insert(buffer, '0')
      n = n * 2
    end
  end
  return table.concat(buffer) .. ' * 2^' .. exponent
end
]]
eval(code)
return code ?>

And so, let's test convert 9.25: <?return convert_to_floating_point(9.25, 9)?>

Which makes sense, as 9.25 = 9 + 1/4; in binary: 1001 + 1/100 = 1001.01

Note that since scientific binary numbers always begin with "1." (except for
the zero), ieee-754 floats ignore this "1." when storing them as it becomes
redundant.  What they do contain is the following:

  * The sign bit (0 for positive, 1 for negative)
  * Exponent bits:
      We don't need to know how these bits are used, but in case you are
      curious:
      We first add a constant to the exponent called "exponent bias", this bias
      depends on the size of this field:
      on 32bit floats the exponent has only 8 bits and the bias is 127, while
      on 64bit floats the exponent has 11 bits with 1023 as the bias.
      Then we clamp its value to the maximum and minimum representable
      exponents: if it ends up at zero we shift the mantissa to the right by
      how much we went below zero to represent subnormal numbers, and if it
      went beyond the max value we store all ones in the mantissa as well to
      represent ±infinity.
  * Mantissa bits:
      As we have mentioned we only keep the fractional part, padding with
      zeros at the end, that's it.

Since in 64bit floats we have 52 bits to store the fractional part, this means
that any bits to the right of those 52 bits will be lost.

Moreover particular the precision of the floating point number will depend on
the size of the integer part!, that would be the number of seconds from epoch,
and so, these are the date ranges for which we would expect a given precision:
<?

-- we will not run this code here as it will get truncated at 2038!, yes,
-- this machine runs on a 32 bits OS which is prone to the y2k38 issue 🤦‍
-- so the output was the result of running this code manually on a 64bits cpu
if false then

function prec(t)
  for e=1, 100 do
    if t==t+2^-(e+1) then return 2^-e end
  end
end

function isodate(t)
  local f = os.date('!*t', t)
  return ('%04d-%02d-%02d %02d:%02d:%02d'):format(
    f.year, f.month, f.day, f.hour, f.min, f.sec)
end

local out = ''
for i=1,52 do
  print(isodate(2^i) .. ' .. ' .. isodate(2^(i+1)-1) .. (' -> %5g'):format(prec(2^i)))
  -- out = out .. (isodate(2^i) .. ' .. ' .. isodate(2^(i+1)-1) .. (' -> %5g'):format(prec(2^i))) .. '\n'
end
return out

end
?>
1970-01-01 00:00:02 .. 1970-01-01 00:00:03 -> 4.44089e-16
1970-01-01 00:00:04 .. 1970-01-01 00:00:07 -> 8.88178e-16
1970-01-01 00:00:08 .. 1970-01-01 00:00:15 -> 1.77636e-15
1970-01-01 00:00:16 .. 1970-01-01 00:00:31 -> 3.55271e-15
1970-01-01 00:00:32 .. 1970-01-01 00:01:03 -> 7.10543e-15
1970-01-01 00:01:04 .. 1970-01-01 00:02:07 -> 1.42109e-14
1970-01-01 00:02:08 .. 1970-01-01 00:04:15 -> 2.84217e-14
1970-01-01 00:04:16 .. 1970-01-01 00:08:31 -> 5.68434e-14
1970-01-01 00:08:32 .. 1970-01-01 00:17:03 -> 1.13687e-13
1970-01-01 00:17:04 .. 1970-01-01 00:34:07 -> 2.27374e-13
1970-01-01 00:34:08 .. 1970-01-01 01:08:15 -> 4.54747e-13
1970-01-01 01:08:16 .. 1970-01-01 02:16:31 -> 9.09495e-13
1970-01-01 02:16:32 .. 1970-01-01 04:33:03 -> 1.81899e-12
1970-01-01 04:33:04 .. 1970-01-01 09:06:07 -> 3.63798e-12
1970-01-01 09:06:08 .. 1970-01-01 18:12:15 -> 7.27596e-12
1970-01-01 18:12:16 .. 1970-01-02 12:24:31 -> 1.45519e-11
1970-01-02 12:24:32 .. 1970-01-04 00:49:03 -> 2.91038e-11
1970-01-04 00:49:04 .. 1970-01-07 01:38:07 -> 5.82077e-11
1970-01-07 01:38:08 .. 1970-01-13 03:16:15 -> 1.16415e-10
1970-01-13 03:16:16 .. 1970-01-25 06:32:31 -> 2.32831e-10
1970-01-25 06:32:32 .. 1970-02-18 13:05:03 -> 4.65661e-10
1970-02-18 13:05:04 .. 1970-04-08 02:10:07 -> 9.31323e-10
1970-04-08 02:10:08 .. 1970-07-14 04:20:15 -> 1.86265e-09
1970-07-14 04:20:16 .. 1971-01-24 08:40:31 -> 3.72529e-09
1971-01-24 08:40:32 .. 1972-02-16 17:21:03 -> 7.45058e-09
1972-02-16 17:21:04 .. 1974-04-03 10:42:07 -> 1.49012e-08
1974-04-03 10:42:08 .. 1978-07-04 21:24:15 -> 2.98023e-08
1978-07-04 21:24:16 .. 1987-01-05 18:48:31 -> 5.96046e-08
1987-01-05 18:48:32 .. 2004-01-10 13:37:03 -> 1.19209e-07
2004-01-10 13:37:04 .. 2038-01-19 03:14:07 -> 2.38419e-07
2038-01-19 03:14:08 .. 2106-02-07 06:28:15 -> 4.76837e-07
2106-02-07 06:28:16 .. 2242-03-16 12:56:31 -> 9.53674e-07
2242-03-16 12:56:32 .. 2514-05-30 01:53:03 -> 1.90735e-06
2514-05-30 01:53:04 .. 3058-10-26 03:46:07 -> 3.8147e-06
3058-10-26 03:46:08 .. 4147-08-20 07:32:15 -> 7.62939e-06
4147-08-20 07:32:16 .. 6325-04-08 15:04:31 -> 1.52588e-05
6325-04-08 15:04:32 .. 10680-07-14 06:09:03 -> 3.05176e-05
10680-07-14 06:09:04 .. 19391-01-25 12:18:07 -> 6.10352e-05
19391-01-25 12:18:08 .. 36812-02-20 00:36:15 -> 0.00012207
36812-02-20 00:36:16 .. 71654-04-10 01:12:31 -> 0.000244141
71654-04-10 01:12:32 .. 141338-07-19 02:25:03 -> 0.000488281
141338-07-19 02:25:04 .. 280707-02-04 04:50:07 -> 0.000976563
280707-02-04 04:50:08 .. 559444-03-08 09:40:15 -> 0.00195313
559444-03-08 09:40:16 .. 1116918-05-14 19:20:31 -> 0.00390625
1116918-05-14 19:20:32 .. 2231866-09-25 14:41:03 -> 0.0078125
2231866-09-25 14:41:04 .. 4461763-06-20 05:22:07 -> 0.015625
4461763-06-20 05:22:08 .. 8921556-12-07 10:44:15 -> 0.03125
8921556-12-07 10:44:16 .. 17841143-11-13 21:28:31 -> 0.0625
17841143-11-13 21:28:32 .. 35680317-09-25 18:57:03 -> 0.125
35680317-09-25 18:57:04 .. 71358665-06-19 13:54:07 ->  0.25
71358665-06-19 13:54:08 .. 142715360-12-06 03:48:15 ->   0.5
142715360-12-06 03:48:16 .. 285428751-11-12 07:36:31 ->   1

In other words, you will be/were able to have:
  nanoseconds precision until 1970-04-08
  microseconds precision until 2242-03-16
  milliseconds precision until 280707-02-04
  and seconds precision until 285428751-11-12

<?

    return title 'SUMMARY'

?>

If you want to store the time in a 64bits floating point number, remember:

  * You can relatively safely use it to store microseconds with perfect
    precision until 2242-03-16, but it will start skipping microseconds
    from that date (probably not an issue right now).

  * It's totally safe to store times with milliseconds precision.
    I don't think that any of the hardware that we currently have or run
    will be able to reach the year 280707.

Postscript:

  If you want to know who uses this format, take a look at the following links:

  https://w3.impa.br/~diego/software/luasocket/socket.html#gettime
  https://docs.python.org/3/library/time.html#time.time
  https://docs.ruby-lang.org/en/master/Time.html#method-i-to_f
  https://www.postgresql.org/docs/current/functions-datetime.html#FUNCTIONS-DATETIME-EXTRACT
    * try: select pg_typeof(extract(epoch from now())); -- if you don't believe

Suggested material for reading on the topic:
  https://www.leebutterman.com/2021/02/01/store-your-unix-epoch-times-as-float64.html
  https://en.wikipedia.org/wiki/IEEE_754
  https://en.wikipedia.org/wiki/Time_formatting_and_storage_bugs
  https://en.wikipedia.org/wiki/Year_2038_problem
  https://ocw.mit.edu/courses/6-00-introduction-to-computer-science-and-programming-fall-2008/resources/lecture-5/
    * this is an introduction to floating point numbers in python.  I add this
      link because the other videos in the course are also interesting for
      anyone starting with programming, even though it's a bit dated by now.

<? return utterances() ?>
[cancel] password:  [private]