[LinuxPPS] ntptime status - relaunch of that issue

[Hal V. Engel]

On Thursday 19 March 2009 09:30:39 am Felix Joussein wrote:
> Hello Hal,
>
> I have just been spending the last 6 hours, doing some short-term tests
> - long-term tests have been running the passt few weeks - with my
> machines, and wanted to let you know my status:
>
> I performed the tests using my stable 2.4er based setup and my 2.6er setup.
> As Hardware I used an Intel-Atom-, Intel Pentium-M and Intel Pentium 4
> system.
>
> The results:
> 2.4er setup is running stable with offset/jitter +/- 0.010 an all
> machines, synchronizing within 10 minutes, stratum status is lowering
> down from 16 to 1 as the offset/jitter comes within normal parameters.
> On H-T cap. CPU's I had to disable it via bios, else my keyboard was
> locked (tried that several times on all H-T architectures - same effect.).
>
> 2.6er setup is also running stable, H-T seams not to be an issue, but
> the Stratum problem, as well as the eternal long synchronizing time from
> about 8 hours (info from my college)  makes it unusable for productive
> environments.
>
> Further my college has noticed, that environmental change (Temperature)
> has a direct impact on the stability.
>
> As you have mentioned, that your setup does meet pretty much what I'm
> missing on mine: quick syncing, stratum level only 1 when it's synced
> etc...: can you tell me, what hardware, which kernel version eventually
> even attach  it's .config file, the cat /proc/cmdline , and your ntpd
> Version + configure options?

My ntp machine is a amd64 4800+ X2 machine with an nvidia nforce 4.x chip 
set/BIOS.  This CPU is not tsc invariant.  The motherboard is unmodified.  The 
machine is fairly loaded with 3 disk drives and a hot high end graphics card 
(about 80 watts under load and 30 watts idle) with an after market heat sink.   
I am currently running kernel 2.6.26 with the 2.6.26 LinuxPPS patch set.  NTP 
is currently a recent development release (4.2.5_p157) but I have also used 
the most recent stable release (4.2.4_p6) as well with the same basic results.  
I did have issues with 4.2.4_p4 however.

Kernel config parms that might affect this:

$ cat /usr/src/linux/.config | grep TIME
CONFIG_GENERIC_TIME=y
CONFIG_GENERIC_TIME_VSYSCALL=y
# CONFIG_KTIME_SCALAR is not set
CONFIG_TIMERFD=y
CONFIG_HIGH_RES_TIMERS=y
CONFIG_HPET_TIMER=y
CONFIG_X86_PM_TIMER=y
CONFIG_NETFILTER_XT_MATCH_TIME=m
CONFIG_SERIAL_8250_RUNTIME_UARTS=5
CONFIG_HANGCHECK_TIMER=m
# CONFIG_PPS_CLIENT_KTIMER is not set
CONFIG_SND_TIMER=y
# CONFIG_SND_RTCTIMER is not set
CONFIG_LEDS_TRIGGER_TIMER=m
# CONFIG_PRINTK_TIME is not set

$ cat /usr/src/linux/.config | grep PPS
# PPS support
CONFIG_PPS=m
CONFIG_PPS_IRQ_EVENTS=y
# CONFIG_PPS_DEBUG is not set
# PPS clients support
# CONFIG_PPS_CLIENT_KTIMER is not set
CONFIG_PPS_CLIENT_LDISC=m
# CONFIG_PPS_CLIENT_LP is not set

$ cat /usr/src/linux/.config | grep SERIAL
CONFIG_PARPORT_SERIAL=y
CONFIG_MOUSE_SERIAL=m
# CONFIG_SERIAL_NONSTANDARD is not set
CONFIG_SERIAL_8250=y
# CONFIG_SERIAL_8250_CONSOLE is not set
CONFIG_SERIAL_8250_PCI=y
CONFIG_SERIAL_8250_PNP=y
CONFIG_SERIAL_8250_NR_UARTS=5
CONFIG_SERIAL_8250_RUNTIME_UARTS=5
CONFIG_SERIAL_8250_EXTENDED=y
CONFIG_SERIAL_8250_MANY_PORTS=y
CONFIG_SERIAL_8250_SHARE_IRQ=y
# CONFIG_SERIAL_8250_DETECT_IRQ is not set
# CONFIG_SERIAL_8250_RSA is not set
CONFIG_SERIAL_CORE=y
# CONFIG_SERIAL_JSM is not set
# CONFIG_SND_SERIAL_U16550 is not set
# CONFIG_USB_SERIAL is not set

$ cat /usr/src/linux/.config | grep HZ
# CONFIG_NO_HZ is not set
CONFIG_HZ_100=y
# CONFIG_HZ_250 is not set
# CONFIG_HZ_300 is not set
# CONFIG_HZ_1000 is not set
CONFIG_HZ=100

I am forcing the HPET timer during startup.  My HPET timer runs at 25MHz.

dmesg | grep hpet
hpet clockevent registered
hpet0: at MMIO 0xfed00000, IRQs 2, 8, 31
hpet0: 3 32-bit timers, 25000000 Hz

I run a Gentoo box and ntp was built using portage with USE=parse-clocks which 
enables all ref clock drivers as well as SHMEM.  The 4.2.5_p157 ebuild is from 
the jhcloss overlay.

Before using --panicgate when starting ntp it used to take my machine several 
hours to stabilize after a restart.  Now that I am using --panicgate it starts 
out with relatively small offsets and converges fairly quickly.

One of the differences between the 2.4 kernel patch set (PPSKIT I think) and 
LinuxPPS is that the 2.4 patches had a kernel consumer and LinuxPPS does not.  
This allows the kernel to adjust the PLL time constant which means that it can 
use a smaller time constant when the oscillator is less stable (IE. changing 
temperatures) and longer time constants when the oscillator is more stable.  
Without the kernel consumer ntp uses a more or less fixed PLL time constant 
which is fairly long.  As a result it reacts slowly to fast temperature 
changes where as kernels with an in kernel consumer (PPSKIT and FreeBSD for 
example) will tend to react to things like this more quickly.  

A few days ago I was in Fry's electronics and while I was there I spent a few 
minutes looking at various motherboards.  What I was interested in was the 
location of the oscillator crystal(s).  Most motherboards had at least 2 
crystals (one of these was located near the USB and firewire controller and 
was probably for those devices) and some had as many as 4.  In most cases the 
main oscillator crystal for the CPU and various timers was located near the 
main motherboard chip set.  In the case of the Atom it was located right next 
to the chip set heat sink and was almost touching the heat sink.  On the Atom 
motherboard the big heat sink with the fan is for the chip set which produces 
significantly more heat that the CPU which does not have a fan on it's 
smallish heat sink.  But having the oscillator crystal near the main source of 
heat on the motherboard has to have an impact on how quickly it's temperature 
"stabilizes" (IE. gets close to it normal operating temperature range) and how 
fast the temperature is changing during the period right after startup. This 
might be one of the reasons that this machine takes so long to stabilize the 
time.   I noticed that a number of motherboards have the crystal very near the 
main motherboard chip set (say less than 1 cm) but that others had this 
located well away from the chip set and other heat sources (IE. > 1 cm).  My 
motherboard for example has this located about 4 cm from the nearest 
motherboard heat sink but right behind the graphics card slot which means that 
the main internal influence on it's temperature is the GPU heat sync.

Although in an ideal world the crystal would be a constant temperature we 
don't live in an ideal world.  The main issue is not that the temperature of 
the crystal is changing.  The real issue is how fast the temperature is 
changing because the long ntp PLL time constant limits how quickly ntp can 
respond to these changes and if the temperature changes too quickly then ntp 
ends up playing catching up because it can't handle how fast the oscillator is 
drifting.  So anything you can do to reduce how quickly the temperature of the 
crystal changes will help.   Remco has actually built a thermostatically 
controlled heater which he connected to the crystal on one of his machines and 
even with a kernel consumer (he was running FreeBSD) he saw a significant 
improvement in his time keeping.  Search the list for his posts about this.

My point is that both the startup issues and temperature issues have been 
topics both on this list and on other timekeeping related forums as these are 
issues for users of reference clocks and ntp on all platforms.  I was able to 
"fix" the startup issue by using --panicgate and I think this should work for 
most machines.  The temperature issues is harder.  I have located what I think 
is the main crystal on my motherboard and at this point I am about to try 
adding heat insulation around the crystal with the hope that this would reduce 
how fast the temperature of the crystal changes and that this might reduce 
temperature induced time keeping errors.  But at this point I don't know how 
much impact this will have if any.  But this will be a cheap experiment so I 
think it is worth trying.

>
> Thank you one more time for helping me!
>
> regards,
>
> Felix
>
> Hal V. Engel schrieb:
> > On Thursday 05 March 2009 02:38:35 am Felix Joussein wrote:
> >> Big thank to all of you who have helped to track down the problem!
> >>
> >>
> >> The best results seam to run on an intel atom (330) with hped
> >> clocksource.
> >>
> >> Now there is one more little issue:
> >> 1. How do I get hped as default timer at boot time?
> >
> > Add
> >
> > hpet=force
> >
> > to your boot command.
> >
> >> When booting my machine it's always tsc, which brings worse results.
> >> 2. An other issue is the time ntpd needs to come down to an offset of
> >> max 0.010 and an jitter of max 0.003.
> >> On my machine it takes about 4-5 hours...
> >
> > I have added -g (--panicgate) to the list of parms that is used to start
> > ntp. On my machine this is set in /etc/conf.d/ntpd but it may be
> > different depending on what distro you are using.  What this does is
> > forces a one time syncing of the system clock to the refclock (IE. a jump
> > in the time) when ntpd is started.  I believe that by default this does
> > not happen unless the clock is off by more than 128 milliseconds.  But I
> > could be wrong about this and you can adjust this using the -x (--slew)
> > switch.
> >
> > My clock usually drifts by more than 128 milliseconds during down times
> > (like over night).  With --panicgate I start out with an offset < 50
> > microseconds (IE. 0.050 max) and things settle down to the 5 to 10
> > microsecond range in perhaps 15 or 20 minutes as the temperature of the
> > machine stabilizes.
> >
> > In my book an offset of 50 microseconds means that my machine is a true
> > stratum 1 machine almost as soon as ntp is running and has reached the
> > refclock which takes less than a minute.  Also with --panicgate ntp will
> > not report having reached the refclock until it has synced with it and
> > made the initial larger time adjustment.
> >
> > One other consideration.  If you have more than one server line in your
> > ntp configuration and you have --panicgate set then ntp will sync the
> > clock with which ever server it first connects with.  The concern is that
> > you may want it to only sync to the local refclock but it may connect to
> > one of the other (less accurate) servers first and use it to make this
> > initial adjustment resulting in a larger initial offset than you want.
> >
> >> it starts at about 40.000 /
> >> 0.800. The 4-5 hours wouldn't be that much of a problem,
> >
> > This is what my machine was like before I started using ---panicgate when
> > starting ntp.
> >
> >> fare more the
> >> fact, that ntpd switches to stratum1 eventhought it's not as precise as
> >> desired... So the second question is: can I set a threshold which limits
> >> the
> >> stratum1 to a defined offset/jitter level?
> >
> > I just did a man ntpd and didn't see anything.  The only threshold
> > adjustment I saw was the slew/jump threshold that can be changed with -x
> > or --slew.  But this will not do what you are asking about.  NTP is
> > intended for time servers that are expected to run continuously so the
> > design is not intended to cause a quick clock sync.  So it does have some
> > issues on machines that are rebooted or have down time like a
> > workstation/end user machine.   Your best bet is to try --panicgate.
> >
> >> 3. Can I accelerate the time adjusting from the pps (offset/jitter)
> >> signal?
> >
> > See above.
> >
> >> 4. More hardware: has anyone of you experience with VIA C7 CPU's and the
> >> clocksource issue?
> >>
> >> 5. anybody at CeBIT that weekend?
> >> Regards,
> >>
> >> Felix
> >>
> >> Hal V. Engel schrieb:
> >>> On Tuesday 24 February 2009 02:52:23 am Felix Joussein wrote:
> >>>> Hello Michael,
> >>>>
> >>>> I have checked that the current clock source is tsc and that 3
> >>>> different are available in total: tsc, acpi_pm and jiffies
> >>>
> >>> Have you enables HPET in the kernel config?
> >>>
> >>> Also many motherboards have spread spectrum oscillators with introduces
> >>> continuous variations into the frequency of the system oscillator. 
> >>> This is intended to reduce radio frequency interference by spreading
> >>> the spurious RF over a wider frequency range so that the MB can meet
> >>> regulatory requirements. But it can cause issues for accurate time
> >>> keeping.  In some cases the BIOS settings will allow you to turn off
> >>> this "feature" so it is worth the effort to check to see if you can do
> >>> this.
> >>>
> >>>> I have changed them to see what happens:
> >>>>
> >>>> tsc & acpi_pm show no differences...
> >>>> jiffies let's the beeper beep permanently... as soon as the keyboard
> >>>> produces a short beep (for ex. backspace on begin of line)... and
> >>>> after setting jiffies you can't reset it to tsc (the echo tsc >
> >>>> /sys/.....) is useless... not exewcuted, because a cat of the file
> >>>> still shows jiffies.
> >>>>
> >>>> I also have compared the behavior  from my former 2.4 installation
> >>>> mentioned in my posts before, and realized, that the ntpd would not
> >>>> even start.
> >>>> In dmesg I get: many lines "do_clock_gettime: negative time wrap on
> >>>> CPU#0: -60751978ns (0,7646704)" and ntpd produces
> >>>> "adjtimex: ntpd could be using obsolete ADJ_TICKADJ" and then stops
> >>>> itself again.
> >>>>
> >>>> The ppstest equivalent in 2.4 is ppsctl, the output is pretty similar
> >>>> to ppstest, the jitter is bouncing from 640 to -32, -672, 704 per
> >>>> line.
> >>>>
> >>>> when I try to run ntpd -d I get a lot debugging output, and at the end
> >>>> a freezed screen. (So definitely this hardware s***ks, maybe I better
> >>>> turn it into a DVR device!)
> >>>>
> >>>> The current machines, running 2.4 kernel + PPSKit are Pentium III with
> >>>> 512MB Ram...
> >>>
> >>> The PIII has an invariant TSC counter and that would explain at least
> >>> part of it's good time keeping performance.
> >>>
> >>>> The Idea to use Pentium M's was to limit power usage.... the
> >>>> industrial PC I'm using now has a Pentium M and 2 GB RAM is using 80W
> >>>> at most... So: how about an Intel ATOM?
> >>>
> >>> Maybe but there was a long thread about TSC counters on the ntp mailing
> >>> list back in the time frame after AMD had announced their plan to make
> >>> future AMD CPUs TSC invariant and there appeared to be skepticism at
> >>> that time about Intel's offerings in this regard at least in part
> >>> because the feature was poorly documented.
> >>>
> >>>> So, what would be the recommended hardware setup?
> >>>
> >>> Another option might be to use one of the low power amd64 CPUs that was
> >>> recently released.  I know that one of these is rated at 9 watts
> >>> (single core, there is also a dual core unit rated at 22 watts) and
> >>> with a low power consumption MB chip set (AMD makes some of these) you
> >>> should end up with a system with very low over all power consumption
> >>> that is in the same ball park as an Atom system (the chip set used on
> >>> the Atom MB has fairly high power consumption and this is the chip with
> >>> the big heat sink on it) and you will almost for sure end up with a
> >>> solidly implemented invariant TSC.
> >>>
> >>> I should add that it is almost impossible to know how good any hardware
> >>> setup will work without first testing it.   But if I was building a new
> >>> system that was intended as a low power consumption time keeping
> >>> network appliance I would go with one of these low power AMD CPUs if
> >>> for no other reason than that the TSC features are well documented and
> >>> were specifically intended for time keeping.
> >>>
> >>>> thank's for helping me!
> >>>>
> >>>> regards,
> >>>>
> >>>> Felix Joussein
> >>>>
> >>>> Michael Meier schrieb:
> >>>>>> I played around with different hardware and realised, that then
> >>>>>> faster a machine is, then less the offset / jitter is.
> >>>>>> For example on an amd64 X2 5000+ jitter/offset is about 0,100 -
> >>>>>> 0,010 +/- while with the Pentium M processor the offset/jitter rises
> >>>>>> up far beyond 1,000...
> >>>>>> I think, the Pentium M (1700MHz) should be way enough for a ntp
> >>>>>> server?
> >>>>>
> >>>>> The CPU Power surely is enough, but you might hit an architectural
> >>>>> problem with the Pentium M: It has a timestamp counter [1] that does
> >>>>> NOT increase monotonically, even when all power saving features are
> >>>>> disabled. Since the TSC is the default and preferred timesource of
> >>>>> the kernel, that will cause some confusion. The kernel will detect
> >>>>> that the TSC is crap after some time, and switch to another source
> >>>>> after some time if it has some available, but the TSC really is the
> >>>>> best you can get. The only thing that comes close to it is HPET (High
> >>>>> Precision Event Timer), and that is not available everywhere. And
> >>>>> naturally, if ntpd cannot get the local time from the kernel with a
> >>>>> sufficiently high resolution, things will start to jitter.
> >>>>> You should check what your kernel currently uses as timesource (cat
> >>>>> /sys/devices/system/clocksource/clocksource0/current_clocksource),
> >>>>> and what sources it has available
> >>>>> (/sys/devices/system/clocksource/clocksource0/current_clocksource).
> >>>>> You can also check in the kernel logs, the kernel usually tells you
> >>>>> when it changes the timesource and why.
> >>>>>
> >>>>> [1] http://en.wikipedia.org/wiki/Time_Stamp_Counter
> >>>>
> >>>> _______________________________________________
> >>>> LinuxPPS mailing list
> >>>> LinuxPPS at ml.enneenne.com
> >>>> http://ml.enneenne.com/cgi-bin/mailman/listinfo/linuxpps
> >>>> Wiki: http://wiki.enneenne.com/index.php/LinuxPPS_support
> >>>
> >>> _______________________________________________
> >>> LinuxPPS mailing list
> >>> LinuxPPS at ml.enneenne.com
> >>> http://ml.enneenne.com/cgi-bin/mailman/listinfo/linuxpps
> >>> Wiki: http://wiki.enneenne.com/index.php/LinuxPPS_support
> >
> > _______________________________________________
> > LinuxPPS mailing list
> > LinuxPPS at ml.enneenne.com
> > http://ml.enneenne.com/cgi-bin/mailman/listinfo/linuxpps
> > Wiki: http://wiki.enneenne.com/index.php/LinuxPPS_support
>
> _______________________________________________
> LinuxPPS mailing list
> LinuxPPS at ml.enneenne.com
> http://ml.enneenne.com/cgi-bin/mailman/listinfo/linuxpps
> Wiki: http://wiki.enneenne.com/index.php/LinuxPPS_support