promofaux / matterfeed.net

Parse various Feeds and post them to your Mattermost server!

👋

I am Adam - I don't really like writing bios.

I am on Twitter, where I mostly waffle on about nothing into the void.

I am on my website, where I also waffle on about nothing into the void, but in longer form.

I am on this website (right here, where you're looking), where I write code for myself and for Pi-hole.

I am on Instagram, where I sometimes post pictures of things I like.

I am on email, because you simply have to be these days.

I am on Linkedin, which I regret every time I visit it.

matterfeed.net's Issues

Inability to resolve a feed causes crash in the program

Using pre-built container promofaux/mattermostrss
following the container logs (docker logs -f MattermostRSS)
this is what I'm seeing before the container restarts:

Unhandled Exception: System.AggregateException: One or more errors occurred. (An error occurred 
while sending the request.) ---> System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.Http.CurlException: Couldn't connect to server
   at System.Net.Http.CurlHandler.ThrowIfCURLEError(CURLcode error)
   at System.Net.Http.CurlHandler.MultiAgent.FinishRequest(StrongToWeakReference`1 easyWrapper, CURLcode messageResult)
   --- End of inner exception stack trace ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at System.Net.Http.HttpClient.<FinishSendAsyncBuffered>d__58.MoveNext()
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at MattermostRSS.Program.GetRssFeed(String url) in /source/MattermostRSS/Program.cs:line 130
   at MattermostRSS.Program.ProcessRssFeeds(RssFeed rssFeed) in /source/MattermostRSS/Program.cs:line 71
   at MattermostRSS.Program.Main(String[] args) in /source/MattermostRSS/Program.cs:line 48

This also means the most recent post from RSS is re-posted in the mattermost channel leading to the channel endlessly filling with repeated content.

Feed failing (silently)

Testing with the following config :

{
"BotCheckIntervalMs": 30000,
"MattermostWebhookUrl": "https://domain/hooks/m4cckh6a4pg",
"BotChannelDefault": "testing",
"BotNameDefault": "bot",
"BotImageDefault": "https://domain/icon.png",
"RssFeeds": [
{
"FeedPretext": "Schneier on Security",
"Url": "https://www.schneier.com/blog/atom.xml",
"BotChannelOverride": "",
"BotNameOverride": "Bruce Schneier",
"BotImageOverride": "https://www.schneier.com/images/bruce-blog3.jpg",
"IncludeContent": false
},
{
"FeedPretext": "Threat post",
"Url": "https://threatpost.com/feed/",
"BotChannelOverride": "",
"BotNameOverride": "Threat post",
"BotImageOverride": "https://threatpost.com/wp-content/themes/threatpost/images/site-icon.png",
"IncludeContent": false
},
{
"FeedPretext": "Blog",
"Url": "https://feeds.feedburner.com/TroyHunt",
"BotChannelOverride": "",
"BotNameOverride": "Troy Hunt",
"BotImageOverride": "https://www.troyhunt.com/favicon.png",
"IncludeContent": false
},
{
"FeedPretext": "Krebs on Security",
"Url": "https://krebsonsecurity.com/feed/",
"BotChannelOverride": "",
"BotNameOverride": "Brian Krebs",
"BotImageOverride": "https://krebsonsecurity.com/wp-content/uploads/2009/11/krebsinbelgium.png",
"IncludeContent": false
},
{
"FeedPretext": "Naked Security",
"Url": "https://nakedsecurity.sophos.com/feed/",
"BotChannelOverride": "",
"BotNameOverride": "Sophos",
"BotImageOverride": "https://pbs.twimg.com/profile_images/667033512403476480/HPSTaLj2_200x200.png",
"IncludeContent": false
},
{
"FeedPretext": "Bulletins",
"Url": "https://www.auscert.org.au/rss/bulletins/",
"BotChannelOverride": "",
"BotNameOverride": "AusCERT",
"BotImageOverride": "https://pbs.twimg.com/profile_images/476961139285184513/cws56Xhp.png",
"IncludeContent": false
}
],
"RedditJsonFeeds": null
}

I've concluded that all feeds read correctly except the last in the list - no errors are produced unfortunately. All other feeds correctly read and append a "LastProcessedItem" flag/timestamp.
I did note however inconsistency in the "Z" on the end of the timestamp, sometimes it had a preceding space and I was unsure of the reason for that character having been appended.

(Non-Fatal) error processing feed

Response from logs as follows:

Feed Title: Troy Hunt's Blog
FeedType: RSS 2.0
Generator: Ghost 1.11

Problem retrieving feed
Exception Message: '', hexadecimal value 0x1F, is an invalid character. Line 1, position 1.
One or more errors occurred. (Object reference not set to an instance of an object.)

This is from the feed https://feeds.feedburner.com/TroyHunt

The feed still seems to process OK however. I've seen a variation of this error which looked more like just "invalid character. Line 1, position 1.
One or more errors occurred. (Object reference not set to an instance of an object.)" ie it didn't state the character that had caused the error.

'Box' prevents notification displaying subjects

Have noted that due to the box-out surrounding the content (not sure of the correct term sorry) mattermost will not correctly add the subject to a notification.

Using https://github.com/PromoFaux/Matterfeed.NET/blob/master/config/secrets.json.sample as an example -

The first feed listed when mattermost displays a notificaiton will read simply "Bruce Schneier - Schneier on Security"

I suggest moving <title> elements outside of the boxed element in order to have it pop up along with FeedPretext & BotNameOverride to make notifications more useful.

Suggestion - Add log trimming to docker-compose

I've modified my docker-compose.yml to add the following:

logging:
  driver: "json-file"
  options:
    max-size: "200k"
    max-file: "10"

some variation on this might be appropriate for the example yml to prevent logs becoming unmanageable (STDERR output isn't capped and bloats the container after long running periods)

RSS Feed Items with a null publish date are reposted constantly.

Related to : codekoenig/SyndicationToolbox#1

If the issue is unsolvable with SyndicationToolbox, I can use FeedReader instead, however in order to use that, then this PR needs to be merged on ReverseMarkdown due to a conflict in the versions of HtmlAgilityPack.

Test