Category: CoolTools

Posted on

Scraping the Teknoids Mailman PiperMail Archive

Putting this here in case anyone finds themselves in need of something to scrape a Pipermail web archive of a Mailman mailing list. This bit of Python 3 is based on a a bit of Python 2 I found at Scraping GNU Mailman Pipermail Email List Archives. The only changes I made from the original are to update somethings to work in Python 3. It works well for my purposes, generating a single text file of the teknoids list archive from 2005 to today.

#!/usr/bin/env python

import requests
from lxml import html
import gzip
from io import BytesIO

listname = 'teknoids'
url = 'https://lists.teknoids.net/pipermail/' + listname + '/'

response = requests.get(url)
tree = html.fromstring(response.text)

filenames = tree.xpath('//table/tr/td[3]/a/@href')

def emails_from_filename(filename):
print (filename)
response = requests.get(url + filename)
if filename[-3:] == '.gz':
contents = gzip.GzipFile(fileobj=BytesIO(response.content)).read()
else:
contents = response.content
return contents

contents = [emails_from_filename(filename) for filename in filenames]
contents.reverse()

contents = b"\n\n\n\n".join(contents)

with open(listname + '.txt', 'wb') as filehandle:
filehandle.write(contents)

Posted on

The WordPress.org Pattern Directory and Creator Now Open to the Public

The WordPress pattern directory finally opened its gates to the entire community today. Anyone with a WordPress.org account can log in, go to the pattern creator, and start designing. For some of u…

Source: The WordPress.org Pattern Creator Now Open to the Public

This looks like a great resource to help those of us who are “design challenged”. Since the patterns are applied at the page level they will help get that final touch in place that is often missing from a theme.

Here’s an example pattern lifted right from the WordPress Patterns directory.

Our Service

Rorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.

Clean Design

Lorem ipsum dolor sit amet, elit, a sed diam nonummy nibh tincidunt consectetuer adipiscing

Read More

Creative Idea

Lorem ipsum dolor sit amet, elit, a sed diam nonummy nibh tincidunt consectetuer adipiscing

Read More

Communication

Lorem ipsum dolor sit amet, elit, a sed diam nonummy nibh tincidunt consectetuer adipiscing

Read More

Patterns are simply copy and pasted from the directory and then edited locally to your needs. Simple.
Posted on

Looking to build an open access repository? EPrints on Ubuntu 20.04 may be just the thing

EPrints is open-source software designed for building open access repositories that are OAI-PMH compliant. It is primarily used for scientific journals and institutional repositories although it can be also used as a document management system.

Source: How To Install EPrints on Ubuntu 20.04 | RoseHosting

Eprints is one of those systems that pops up on my radar every now and then. My favorite things about it is that it’s written in Perl and it’s still in active development. This article is general enough that it will apply to just about any hosting situation.

I’ve tinkered with EPrints and it works as advertised. I suppose if academic law libraries had more support for self hosted applications we might see it in legal academia. You can find the source on Github, more info at EPrint, and try out a demo version.

Posted on

Can I create a syllabus with Node-RED?

Node-RED is a programming tool for wiring together hardware devices, APIs and online services in new and interesting ways. It provides a browser-based editor that makes it easy to wire together flows using the wide range of nodes in the palette that can be deployed to its runtime in a single-click.
Node-RED

Here’s an idea: use the low code browser based flow design tools of Node-RED to create a syllabus of online resources. Since Node-RED can work with APIs and I have an API for CALI resources, I should be able to use Node-RED tools to design a syllabus as a flowchart through a series of resources.

I suspect there are a few missing pieces here like a program or routine to step through the flow created by Node-RED and a system from tracking results that talks to Node-RED. The pieces are likely buildable one identified. The result would be a programmed learning environment using Node-RED as a primary component.