rssed — Become A Developer Beyond Bookmarks
RSS a.k.a Really Simple Syndication is a great technology to subscribe to website content. You can get the latest updates published on a website by parsing the RSS feed. It uses XML to present the data and meta information about the content.
A huge number of software developers have their own blog where they publish posts about programming and software development regularly. Initially, I was bookmarking blogs which I admire and used to visit them once in a while. But that wasn't practical enough. I was missing on beneficial updates and news of what was happening in the software world.
So, I decided to keep track of all of those blogs using a blogroll — a collection of links to other sites or blogs — which updates itself on a daily basis. With the help of an rss parser, astro and a little bit of logic, I present to you rssed.netlify.app and the journey of how I built it.
Table Of Contents
- Initial Idea
- Astro
- RSS Feeds
- Daily Build
- Labelling Latest Posts
- Beautify Logs
- Deploying to Netlify
- Contribute
Initial Idea #
At first, I thought to create a postgres database on neon for storing the feeds urls and I did implement it, but unfortunately, after having some issues when deploying it to a serverless function on vercel, I dropped it.
Went for SSR at first (due to the urge to try dynamic paths in astro), but finally resorted to do everything at build time. Because why not? If you have a list of feeds ready, instead of generating the page from the same template on each feed request again and again, generate all of it beforehand and then serve.
Astro #
To give you some background — astro is a javascript based meta framework for building websites focused around content. You can use it to build static sites as well as interactive web applications using the framework of your choice, however I prefer the former use case much more than the latter. Astro's main focus is on adding interactivity through hydration (sprinkling javascript) and isolating interactive components using the islands architecture.
Astro primarily generates static sites but it also gives you 2 options to build your site output:
server- everything will be built and rendered on demand as per the user request. You can exclude some parts of your site which need to be rendered beforehand. So, here the default is on-demand server rendered content.hybrid- the default here is everything is pre-rendered at build time unless you specify otherwise.
For this blogroll, I went for the hybrid approach as I didn't see an issue generating content at build time. But don't I want to update the feeds on a daily basis? Yes, I still do, and you might be thinking why go for a static approach — more on that later.
There are three .astro files in the picture:
- One for displaying the list of sorted feeds (
src/pages/index.astro) - One for displaying the posts for a single feed (
src/pages/feed/[id].astro) - The third serves as a layout (
src/layouts/Layout.astro) for the previous two.
RSS Feeds #
RSS feeds always fascinate me considering how simple they are. A perfect way to consume content published independently. I was on the search for an RSS parser and eventually found the rss parser by @rbren.
The Web is Fantastic - by Robb Knight
Having stored the feed URLs along with their IDs in a json file, it became easy for me to loop over them and parse them.
import feedlist from "../../data/feedlist.json"
type feedItem = Output<{ [key: string]: any }> & { id: string };
const feeds: {
time: string | null
items: feedItem[]
} = {
time: null,
items: []
}
export const ParseRSS = async (url: string) => {
return await new Parser({
timeout: 120000
}).parseURL(url)
}
const parseAndStoreFeeds = async (list: { id: string, url: string }[]) => {
const feedPromises = list.map(async (site) => {
try {
const feed: feedItem = {
...await ParseRSS(site.url),
id: site.id
}
return feed
} catch (error) {
logger.error({
feed: {
id: site.id,
url: site.url,
},
error
})
}
})
Promise.allSettled(feedPromises)
for await (const feed of feedPromises) {
feed && (
!feeds.items.some(i => i.id === feed.id) && feeds.items.push(feed)
)
}
}
parseAndStoreFeeds(feedlist)What I am doing here is fetching the RSS feeds from their URLs by using the rss-parser parallelly, handling potentially unhandled promise rejections using Promise.allSettled() and pushing them sequentially in an array. This approach of fetching feeds parallelly and storing them sequentially has improved the build time drastically.
Earlier, I was fetching the feeds parallelly but waiting for the last one to get fetched and succeed. But this meant that if one of the URL fails to get parsed, I won't get any result. It's all or nothing. I certainly don't want to do that.
export const allFeeds = async (list: { id: string, url: string }[]) => {
return Promise.all(
list.map(async (site) => {
return {
...await ParseRSS(site.url),
id: site.id
}
})
)
}So, I improved it a bit, but now the problem was that it was all sequential. If one URL takes 6 seconds to resolve and parse, the next one just keeps on waiting. That's too slow.
export const allFeeds = async (list: { id: string, url: string }[]) => {
const feeds = [];
for (const site of list) {
try {
const feed = {
...await ParseRSS(site.url),
id: site.id
}
feeds.push(feed)
} catch (error) {
console.error(error)
}
}
return feeds
}Finally, I remembered I read a post by Jake Archibald discussing the gotcha of unhandled promise rejections and found the solution which I shared at the very first.
const feedPromises = list.map(async (site) => {
const feed: feedItem = {
...await ParseRSS(site.url),
id: site.id
}
return feed
})
// gotcha
Promise.allSettled(feedPromises)
for await (const feed of feedPromises) {
feed && (
feeds.items.push(feed)
)
}Once I receive all of the parsed feeds in index.astro, sorting them according to the publish date of the last post or the last build date keeps them in a descending order.
For displaying posts from a single feed, I went for dynamic routes in astro([id].astro). It's awesome that you can generate multiple pages from the same template by using the getStaticPaths() method and giving it a bunch of values as props with the feed uuid as the param.
export async function getStaticPaths() {
try {
const res = await fetchFeeds();
const { data } = JSON.parse(res);
const feeds = await allFeeds(data as { id: string; url: string }[]);
const feedList: Record<string, Record<string, any>>[] = feeds.map(
(fl) => {
return {
params: {
id: fl.id,
},
props: {
feeds: feeds.filter((f) => f.id === fl.id),
},
};
}
);
return feedList
? feedList
: [
{
params: {
id: "404",
},
props: {
feeds: null,
},
},
];
} catch (error) {
console.log(error);
return [
{
params: {
id: "404",
},
props: {
feeds: null,
},
},
];
}
}And accessing the prop from the Astro.props object.
const { feeds } = Astro.props;Daily Build #
If all of this is static and doesn't even update itself, then what's the point of building a blogroll which subscribes to RSS feeds. That's where Netlify's build hook comes into action. A simple POST request to the build hook URL can trigger a new build of the project. This request is made every day at 00:00 with the help of a cron schedule 0 0 * * *. Learn more about cron timing and play with it on crontab.guru.
The below mentioned code lives under netlify/functions directory.
import fetch from "node-fetch";
import { schedule } from "@netlify/functions";
const BUILD_HOOK = process.env.BUILD_HOOK as unknown as URL
// every day at 00:00
export const handler = schedule('0 0 * * *', async () => {
try {
const res = await fetch(BUILD_HOOK, {
method: "POST"
})
console.log(res)
return {
statusCode: 200
}
} catch (error) {
console.log(error)
return {
statusCode: 500
}
}
})Labelling Latest Posts #
I stored the last updated date as a data- attribute on the feed element, stored the time in localStorage and then accessed that attribute using client side javascript to calculate the difference in time between the localStorage timestamp and last updated time of the feed.
document.addEventListener('DOMContentLoaded', () => {
document.querySelectorAll('.feed').forEach(e => {
const ele = e as HTMLElement;
const feedId = ele.dataset.feedId as string;
const lastPublishedTime = new Date(`${ele.dataset.lastPublished}`).getTime();
if (!localStorage.getItem(feedId)) {
localStorage.setItem(feedId, JSON.stringify({
read: true,
timestamp: lastPublishedTime
}));
ele.classList.remove('feed_banner');
} else {
const prevPostLogRead = JSON.parse(localStorage.getItem(feedId) as string);
const loggedPublishTime = Number(prevPostLogRead.timestamp);
if (!((lastPublishedTime - loggedPublishTime) <= 0)) {
const newPostLogUnread = {
...prevPostLogRead,
read: false
}
localStorage.setItem(feedId, JSON.stringify(newPostLogUnread));
ele.dataset.items === 'true' && ele.classList.add('feed_banner');
} else {
const read = JSON.parse(localStorage.getItem(feedId) as string).read;
read && ele.classList.remove('feed_banner');
}
}
})
})Beautify Logs #
While enhancing some stuff, I thought why not make logs colorful and pretty. And, thus I ended up using consola to beautify the logs.
import { createConsola } from "consola";
const loggerInstance = () => createConsola({
fancy: true,
formatOptions: {
colors: true,
date: true
}
})
export const logger = loggerInstance()Deploying to Netlify #
In order to deploy the astro site with a hybrid or server output mode (server side rendering), you need an adapter. Astro provides official adapters which you can use for deploying to various platforms. Run npm run build locally to see the build output.
import { defineConfig } from 'astro/config';
import netlify from "@astrojs/netlify";
// https://astro.build/config
export default defineConfig({
output: 'hybrid',
server: {
port: 3000
},
adapter: netlify()
});Contribute #
rssed is open source. You are free to contribute either new RSS feeds or code as a developer. I don't know if this has the potential to be the next big project but you can help it be one. Power to you.
GitHub Repository