Mathematics museum that caters heavily to enthusiasts but struggles to welcome curious newcomers into its complex world.