Blog plugin failing with `TypeError: can't compare offset-naive and offset-aware datetimes` #7705

perpil · 2024-11-15T17:34:13Z

Context

I have been successfully building my blog using the blog plugin. I'm not sure whether it was a recent insiders update, daylight savings or some filesystem change but I have not changed my mkdocs.yml file lately. When I attempt to build now or serve, I get this error:

INFO    -  Building documentation...
WARNING -  Debug mode is enabled for "social" plugin.
INFO    -  Cleaning site directory
Traceback (most recent call last):
  File "/opt/homebrew/bin/mkdocs", line 8, in <module>
    sys.exit(cli())
  File "/opt/homebrew/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/opt/homebrew/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/opt/homebrew/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/homebrew/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/homebrew/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/opt/homebrew/lib/python3.10/site-packages/mkdocs/__main__.py", line 272, in serve_command
    serve.serve(**kwargs)
  File "/opt/homebrew/lib/python3.10/site-packages/mkdocs/commands/serve.py", line 85, in serve
    builder(config)
  File "/opt/homebrew/lib/python3.10/site-packages/mkdocs/commands/serve.py", line 67, in builder
    build(config, serve_url=None if is_clean else serve_url, dirty=is_dirty)
  File "/opt/homebrew/lib/python3.10/site-packages/mkdocs/commands/build.py", line 292, in build
    files = config.plugins.on_files(files, config=config)
  File "/opt/homebrew/lib/python3.10/site-packages/mkdocs/plugins.py", line 593, in on_files
    return self.run_event('files', files, config=config)
  File "/opt/homebrew/lib/python3.10/site-packages/mkdocs/plugins.py", line 566, in run_event
    result = method(item, **kwargs)
  File "/Users/david/Documents/GitHub/mkdocs-material/material/plugins/blog/plugin.py", line 134, in on_files
    self.blog.posts = sorted(
TypeError: can't compare offset-naive and offset-aware datetimes

Bug description

The blog plugin appears to choke on the creation dates of some of the files on my filesystem. I'm able to workaround it by modifying line 134 in the blog plugin (material/plugins/blog/plugin.py) from:

        self.blog.posts = sorted(
            self._resolve_posts(files, config),
            key = lambda post: (
                post.config.pin,
                post.config.date.created
            ),
            reverse = True
        )

to (I also add import pytz at the top of the file)

        self.blog.posts = sorted(
            self._resolve_posts(files, config),
            key = lambda post: (
                post.config.pin,
                post.config.date.created.replace(tzinfo=pytz.utc)
            ),
            reverse = True
        )

Reproduction

I was not able to repro this, I tried several things but could not isolate it. The following is the markdown_extensions and plugins sections from the mkdocs.yml where it occurs.

markdown_extensions:
  - attr_list
  - md_in_html
  - pymdownx.emoji:
      emoji_index: !!python/name:material.extensions.emoji.twemoji
      emoji_generator: !!python/name:material.extensions.emoji.to_svg
  - admonition
  - pymdownx.details
  - pymdownx.snippets:
      url_download: true
      restrict_base_path: false
  - pymdownx.superfences
  - pymdownx.tasklist:
      custom_checkbox: true
  - tables
  - footnotes
plugins:
  - meta
  - blog
  - search
  - git-revision-date-localized:
      enable_creation_date: true
      type: timeago
      exclude:
        - blog/*
  - social:
      cards_layout_dir: layouts
      cards_layout: custom
      debug: true
      #debug_on_build: true
  - privacy:
      enabled: true
      cache: true
      cache_dir: .cache/plugins/privacy
      links_attr_map:
        target: _blank
      assets_exclude:
        - platform.twitter.com/widgets.js

And here is the output of my docs/blog directory:

❯ stat -f "%Sc %z %N" *
May 31 15:38:53 2024 4778 2023-year-in-review.md
Oct 28 14:44:33 2024 8815 a-better-readme.md
May 31 15:44:25 2024 7776 bifurcating-lambda-logs.md
May 31 15:44:19 2024 6025 cloudwatch-insights-tricks.md
Aug 14 16:04:14 2024 15969 coldstart-zero-part-duex.md
May 31 15:44:06 2024 9717 coldstart-zero.md
May 31 15:44:01 2024 4142 done-before-the-bass-drops.md
Jun 27 21:06:41 2024 9605 dynamodb-secretmanager.md
Oct 17 12:58:46 2024 8387 edge-metrics.md
May 31 15:44:38 2024 6188 lambda-env-variable-coldstarts.md
May 31 15:44:43 2024 5648 lambda-request-timeline.md
May 31 15:44:52 2024 15597 logging-for-scale.md
Oct 13 17:09:17 2024 988 migrating-to-cdk.md
May 31 15:44:58 2024 2162 mission.md
May 31 15:45:03 2024 23022 optimizing-lambda-coldstarts.md
May 31 15:45:07 2024 8994 stripped-down-coldstarts.md
May 31 15:45:11 2024 5769 the-global-query.md
May 31 15:45:16 2024 4372 totally-async-eventbridge.md
May 31 15:45:21 2024 7874 why-im-spicy-about-coldstarts.md

Steps to reproduce

I tried creating a minimal blog repro both with and without the git-revision-date-localized plugin but was unable to get this issue to reoccur, it only occurs in my main site directory.

plugins:
  - blog
  - git-revision-date-localized:
      enable_creation_date: true
      type: timeago
      exclude:
        - blog/*

I'm using mkdocs-material 9.5.44+insiders.4.53.14

Browser

Firefox

Before submitting

I have read and followed the bug reporting guidelines.
I have attached links to the documentation, and possibly related issues and discussions.
I assure that I have removed all customizations before submitting this bug report.
I have attached a .zip file with a minimal reproduction using the built-in info plugin.

The text was updated successfully, but these errors were encountered:

kamilkrzyskow · 2024-11-15T19:14:47Z

Hello @perpil,
your stat -f "%Sc %z %N" * isn't really relevant here, as the config.date.created is the date in your meta data front-matter, it's not taken from the filesystem.

If you're willing to modify the the Python sources please try this, before the line with the error:

for post in self._resolve_posts(files, config):
    print(post.file.src_uri)
    print(" ", post.meta["date"])
    print(" ", post.config.date.created)

this might show what the input was that caused the error.

I agree with your solution from the reference there needs to be some normalization.
However, pytz is an external dependency. tzinfo=timezone.utc should also work 🤔

perpil · 2024-11-15T19:44:53Z

Here's the output if I put the code you suggest prior to the line that causes the error:

blog/posts/2023-year-in-review.md
  2024-01-04
  2024-01-04 00:00:00
blog/posts/a-better-readme.md
  2023-04-20 00:00:00+00:00
  2023-04-20 00:00:00+00:00
blog/posts/bifurcating-lambda-logs.md
  2024-02-23
  2024-02-23 00:00:00
blog/posts/cloudwatch-insights-tricks.md
  2023-05-11
  2023-05-11 00:00:00
blog/posts/coldstart-zero-part-duex.md
  2024-05-22
  2024-05-22 00:00:00
blog/posts/coldstart-zero.md
  2024-04-19
  2024-04-19 00:00:00
blog/posts/done-before-the-bass-drops.md
  2023-12-12
  2023-12-12 00:00:00
blog/posts/dynamodb-secretmanager.md
  2024-06-27
  2024-06-27 00:00:00
blog/posts/edge-metrics.md
  2024-10-16
  2024-10-16 00:00:00
blog/posts/lambda-env-variable-coldstarts.md
  2024-03-13
  2024-03-13 00:00:00
blog/posts/lambda-request-timeline.md
  2024-02-14
  2024-02-14 00:00:00
blog/posts/logging-for-scale.md
  2023-09-08
  2023-09-08 00:00:00
blog/posts/migrating-to-cdk.md
  2024-10-08
  2024-10-08 00:00:00
blog/posts/mission.md
  2023-04-10
  2023-04-10 00:00:00
blog/posts/optimizing-lambda-coldstarts.md
  2023-09-23
  2023-09-23 00:00:00
blog/posts/stripped-down-coldstarts.md
  2024-02-01
  2024-02-01 00:00:00
blog/posts/the-global-query.md
  2023-11-14
  2023-11-14 00:00:00
blog/posts/totally-async-eventbridge.md
  2023-10-12
  2023-10-12 00:00:00
blog/posts/why-im-spicy-about-coldstarts.md
  2024-01-02
  2024-01-02 00:00:00

Surprisingly, mkdocs serve doesn't fail with those additional prints and no .replace(tzinfo=pytz.utc), it succeeds without error. The code:

        for post in self._resolve_posts(files, config):
            print(post.file.src_uri)
            print(" ", post.meta["date"])
            print(" ", post.config.date.created)
        self.blog.posts = sorted(
            self._resolve_posts(files, config),
            key = lambda post: (
                post.config.pin,
                post.config.date.created
            ),
            reverse = True
        )

Putting: .replace(tzinfo=timezone.utc) fails with:

  File "/Users/david/Documents/GitHub/mkdocs-material/material/plugins/blog/plugin.py", line 142, in <lambda>
    post.config.date.created.replace(tzinfo=timezone.utc)
NameError: name 'timezone' is not defined

kamilkrzyskow · 2024-11-15T19:57:31Z

Thanks for the follow up:

blog/posts/a-better-readme.md
  2023-04-20 00:00:00+00:00
  2023-04-20 00:00:00+00:00

this is the only file with the +00:00, and it appears to be there for post.meta["date"], is the top front-matter in that file any different compared to the other files?

NameError: name 'timezone' is not defined

This needs to be imported via from datetime import timezone, what I meant with pytz, is that it's not included by default during pip install mkdocs-material, so if there is no need let's stick to the built-ins.

perpil · 2024-11-15T20:12:26Z

That was it! I had a date in the frontmatter that was set to: 2023-04-20T00:00:00Z when I removed the timestamp, it worked fine.

Here's a repro:

9.5.44+insiders.4.53.14-blog-plugin-breaks-with-timestamp-frontmatter.zip

I've also confirmed that adding from datetime import timezone fixes the NameError above.

For now I've removed the timestamp from the frontmatter date on the offending file and am unblocked.

squidfunk · 2024-11-15T22:12:23Z

Thanks for reporting. Could you PR the necessary change? If so, please send a PR for the community edition, as I believe all functionality that is impacted was already released.

perpil · 2024-11-16T21:11:30Z

I can do a PR for this change if it's just hardcoding the timezone. A couple of quick questions before I do.

Is the blog plugin the only place this change is necessary?
Is there a better way than hardcoding to get the timezone? Arguably it is user error to include the timestamp and timezone on the date, but iso8601 is a well known format. Is there some way to process the frontmatter so it respects the timezone if it's present, otherwise uses utc?

squidfunk · 2024-11-16T22:09:45Z

Ooph, good questions. So, we can only work with what we get from YAML. We're parsing the front matter here:

mkdocs-material/src/plugins/blog/structure/options.py

Lines 53 to 77 in 596aeea

    
           # Normalize the supported types for post dates to datetime 
        
           def pre_validation(self, config: Config, key_name: str): 
        
               # If the date points to a scalar value, convert it to a dictionary, as 
        
               # we want to allow the author to specify custom and arbitrary dates for 
        
               # posts. Currently, only the `created` date is mandatory, because it's 
        
               # needed to sort posts for views. 
        
               if not isinstance(config[key_name], dict): 
        
                   config[key_name] = { "created": config[key_name] } 
        
               # Convert all date values to datetime 
        
               for key, value in config[key_name].items(): 
        
                   # Handle datetime - since datetime is a subclass of date, we need 
        
                   # to check it first, or we lose the time - see https://t.ly/-KG9N 
        
                   if isinstance(value, datetime): 
        
                       continue 
        
                   # Handle date - we set 00:00:00 as the default time, if the author 
        
                   # only supplied a date, and convert it to datetime 
        
                   if isinstance(value, date): 
        
                       config[key_name][key] = datetime.combine(value, time()) 
        
               # Initialize date dictionary 
        
               config[key_name] = DateDict(config[key_name])

I'm not sure what the best way forward is, but my educated guess (without diving too deep into this topic, as I'm currently fighting other battles) is that if we would support timezones, we should normalize them to UTC, i.e., +00:00 -> Z + adjustments. I'm not too tied up in this topic right now, so I'm open to discussing things in a PR. If somebody wants to go ahead and dive into the reasoning on this one, the help is very much appreciated.

Fixes squidfunk#7705 Normalize datetime values to UTC in blog plugin to handle offset-naive and offset-aware datetimes correctly. * Import `timezone` from `datetime` in `material/plugins/blog/structure/options.py`. * Modify `pre_validation` method to set `tzinfo=timezone.utc` for datetime values. --- For more details, open the [Copilot Workspace session](https://copilot-workspace.githubnext.com/squidfunk/mkdocs-material/issues/7705?shareId=XXXX-XXXX-XXXX-XXXX).

Fixes squidfunk#7705 + Move changes to src directory

squidfunk · 2024-11-20T08:19:00Z

Released as part of 9.5.45.

mvelikikh · 2024-11-20T11:09:13Z

@squidfunk I actually started getting this error with 9.5.45. It works fine with 9.5.38. Looks like the same bug which might be still not fixed. I can open a separate bug if needed.

fails with 9.5.45

pip list | findstr mkdocs-material
mkdocs-material            9.5.45
mkdocs-material-extensions 1.3.1

mkdocs build --strict
INFO    -  Cleaning site directory
INFO    -  Building documentation to directory: C:\some_directory...
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "D:\temp\venv-blog-dev\scripts\mkdocs.exe\__main__.py", line 7, in <module>
  File "D:\temp\venv-blog-dev\Lib\site-packages\click\core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\temp\venv-blog-dev\Lib\site-packages\click\core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "D:\temp\venv-blog-dev\Lib\site-packages\click\core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\temp\venv-blog-dev\Lib\site-packages\click\core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\temp\venv-blog-dev\Lib\site-packages\click\core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\temp\venv-blog-dev\Lib\site-packages\mkdocs\__main__.py", line 288, in build_command
    build.build(cfg, dirty=not clean)
  File "D:\temp\venv-blog-dev\Lib\site-packages\mkdocs\commands\build.py", line 292, in build
    files = config.plugins.on_files(files, config=config)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\temp\venv-blog-dev\Lib\site-packages\mkdocs\plugins.py", line 593, in on_files
    return self.run_event('files', files, config=config)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\temp\venv-blog-dev\Lib\site-packages\mkdocs\plugins.py", line 566, in run_event
    result = method(item, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^
  File "D:\temp\venv-blog-dev\Lib\site-packages\material\plugins\blog\plugin.py", line 133, in on_files
    self.blog.posts = sorted(
                      ^^^^^^^
TypeError: can't compare offset-naive and offset-aware datetimes

works fine with the previous version I had - 9.5.38

pip install mkdocs-material==9.5.38
pip list | findstr mkdocs-material
mkdocs-material            9.5.38
mkdocs-material-extensions 1.3.1

mkdocs build --strict
INFO    -  Cleaning site directory
INFO    -  Building documentation to directory: C:\some_directory
INFO    -  Documentation built in 4.47 seconds

…aterial#7705

squidfunk · 2024-11-20T11:43:53Z

@mvelikikh thanks for reporting. Could you please create a new bug report with a minimal reproduction, as it's related but another error in another location. Maybe @perpil can take a look, as he's the one that fixed the other issue ☺️

kamilkrzyskow · 2024-11-20T12:26:35Z

The error seems to be at the same place, off by one line but I guess this is the difference between the Insiders version.
Looking at the changes that were introduced in the PR, they don't implement the fix proposed in the Issue above.

We discussed the change to replace the datetime object post.config.date.created.replace(tzinfo=timezone.utc)
The PR changed the conversion for the date object:

            if isinstance(value, date):
                config[key_name][key] = datetime.combine(value, time()).replace(tzinfo=timezone.utc)

However, the PR doesn't replace the timezone to utc for the datetime object:

            if isinstance(value, datetime):
                continue

I guess that's the issue, haven't run any tests.

squidfunk · 2024-11-20T12:33:53Z

PRs appreciated!

mvelikikh · 2024-11-20T12:40:47Z

@kamilkrzyskow and @squidfunk, thanks for the replies. I have constructed a minimally reproduced example, and opened a new issue: #7725

It seems that the error is happening only when both of the date created formats are present - it works if I keep only one format:

I constructed a minimal reproduction test case, and based on this, the error is thrown when I have two formats of date.created:

yyyy-mm-dd (e.g. created: 2024-09-06)
yyyy-mm-ddThh:mi:ss (e.g. created: 2019-02-25T04:25:00)
Both formats are supported per the documentation: https://squidfunk.github.io/mkdocs-material/plugins/blog/#meta.date
If I remove any of the files and keep only one format, there is no error.

perpil · 2024-11-20T15:02:14Z

Ah! I thought that by making the change here it would be a more central location, but now I'm realizing the date I was testing with had a Z at the end of it making it timezone aware and the times it is choking on don't have any timezone info. I'll attempt to change it only if it is missing the timezone i.e.

if datetime.tzinfo is None:
   datetime.replace(tzinfo=timezone.utc)

perpil mentioned this issue Nov 18, 2024

Normalize datetime values to UTC in blog plugin #7708

Merged

perpil added a commit to perpil/mkdocs-material that referenced this issue Nov 18, 2024

Normalize datetime values to UTC in blog plugin

615a313

Fixes squidfunk#7705 + Move changes to src directory

squidfunk closed this as completed in #7708 Nov 19, 2024

squidfunk closed this as completed in a08809a Nov 19, 2024

squidfunk added bug Issue reports a bug resolved Issue is resolved, yet unreleased if open labels Nov 19, 2024

squidfunk reopened this Nov 19, 2024

squidfunk closed this as completed Nov 20, 2024

mvelikikh added a commit to mvelikikh/mvelikikh.github.io that referenced this issue Nov 20, 2024

Use a fixed mkdocs-material version to avoid issue squidfunk/mkdocs-m…

c718e2d

…aterial#7705

mvelikikh mentioned this issue Nov 20, 2024

Blog plugin failing with TypeError: can't compare offset-naive and offset-aware datetimes on 9.5.45 #7725

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Blog plugin failing with `TypeError: can't compare offset-naive and offset-aware datetimes` #7705

Blog plugin failing with `TypeError: can't compare offset-naive and offset-aware datetimes` #7705

perpil commented Nov 15, 2024 •

edited

Loading

kamilkrzyskow commented Nov 15, 2024

perpil commented Nov 15, 2024

kamilkrzyskow commented Nov 15, 2024 •

edited

Loading

perpil commented Nov 15, 2024

squidfunk commented Nov 15, 2024 •

edited

Loading

perpil commented Nov 16, 2024

squidfunk commented Nov 16, 2024 •

edited

Loading

squidfunk commented Nov 20, 2024

mvelikikh commented Nov 20, 2024

squidfunk commented Nov 20, 2024

kamilkrzyskow commented Nov 20, 2024

squidfunk commented Nov 20, 2024

mvelikikh commented Nov 20, 2024

perpil commented Nov 20, 2024

Blog plugin failing with TypeError: can't compare offset-naive and offset-aware datetimes #7705

Blog plugin failing with TypeError: can't compare offset-naive and offset-aware datetimes #7705

Comments

perpil commented Nov 15, 2024 • edited Loading

Context

Bug description

Related links

Reproduction

Steps to reproduce

Browser

Before submitting

kamilkrzyskow commented Nov 15, 2024

perpil commented Nov 15, 2024

kamilkrzyskow commented Nov 15, 2024 • edited Loading

perpil commented Nov 15, 2024

squidfunk commented Nov 15, 2024 • edited Loading

perpil commented Nov 16, 2024

squidfunk commented Nov 16, 2024 • edited Loading

squidfunk commented Nov 20, 2024

mvelikikh commented Nov 20, 2024

fails with 9.5.45

works fine with the previous version I had - 9.5.38

squidfunk commented Nov 20, 2024

kamilkrzyskow commented Nov 20, 2024

squidfunk commented Nov 20, 2024

mvelikikh commented Nov 20, 2024

perpil commented Nov 20, 2024

Blog plugin failing with `TypeError: can't compare offset-naive and offset-aware datetimes` #7705

Blog plugin failing with `TypeError: can't compare offset-naive and offset-aware datetimes` #7705

perpil commented Nov 15, 2024 •

edited

Loading

kamilkrzyskow commented Nov 15, 2024 •

edited

Loading

squidfunk commented Nov 15, 2024 •

edited

Loading

squidfunk commented Nov 16, 2024 •

edited

Loading