Posts

Mastodon

 

Deploy a LLM Chat-GPT like service on Jetstream

jetstream
llm
In this tutorial we will deploy a LLM Chat-GPT like service on a GPU node on Jetstream. For experimentation purposes we are using the smaller and cheapest GPU node available…

Kubernetes monitoring with Prometheus and Grafana

kubernetes
jetstream
jupyterhub
In a production Kubernetes deployment it is necessary to make it easier to monitor the status of the cluster effectively. Kubernetes provides Prometheus to gather data from…

 

Generate script to mount a Manila share

kubernetes
jetstream2
Jetstream 2 makes creating volumes that are shared across instances as easy as creating standard volumes. This is provided by the Manila service, which creates volumes based…
Oct 9, 2024

 

Configure ssh-agent on Linux

linux
I keep Googling this again and again.
Oct 2, 2024

 

Tutorial on Python for HPC at the SDSC Summer Institute 2024

python
hpc
Each August the San Diego Supercomputer Center organizes a week-long summer school that teaches a large number of topics related to Supercomputing and AI to early career…
Sep 11, 2024

 

How to use Anki flashcards to help kids learn spelling

documentation
In the Anki Desktop app on your computer, from the main screen, click on the “Create Deck” button to create a new deck, name it “Spelling::Week1” or similar. This is going…
Sep 3, 2024

Generate sample Cosmology data for Cheap and FAIR data portal with PySM

cosmology
healpy
pysm
This notebook generates some figures of Galactic and Extra-Galactic emissions using PySM. Mostly for displaying purposes.
Aug 28, 2024

 

Cosmic Microwave Background data licensing

data
Some thoughts on how to license and attribute data products from the Simons Observatory.
Aug 4, 2024

 

Export a Confluence page to Markdown

documentation
I feel the pain, you have a page trapped in Confluence and you want to set it free. Of course the “Export” features of Confluence are useless.
Aug 4, 2024

 

Ubuntu 22.04 Minimal on Jetstream

kubernetes
jetstream2
Update July 2024: Ubuntu 22.04 Minimal image with Generic Kernel
Jul 31, 2024

Generate point source maps with pixell and convert to HEALPix

cosmology
pysm
Testing the pixell sim_objects functionality to create maps of point sources pre-smoothed with a gaussian beam. The purpose is to include this functionality in PySM to be…
Jul 29, 2024

Run (part of) healpy in the browser with pyodide

python
healpy
Thanks to the help of @VeerioSDSC and Rick Wagner, I have been able to run part of healpy in the browser. For now only tested reading FITS maps and plotting with projview.
Jul 17, 2024

 

Conditional authentication on a single endpoint with FastAPI

python
Building on my previous tutorial, I have now implemented a toy FastAPI application which requires authentication based on query parameters.
Jun 27, 2024

 

Deploy a web server with automatic HTTPS for static website and Django app on Jestream 2

jetstream2
As part of my work to support the Jetstream 2 Openstack deployment at Indiana University, I have developed a tutorial that explains how to create a Virtual Machine that…
Jun 13, 2024

 

Authenticate FastAPI endpoints with a Github organization

python
FastAPI is a framework that simplifies building APIs in Python. Authentication via Google, Github and many more is provided by FastAPI-SSO.
May 29, 2024

 

Jupyter Notebook frontmatter for Quarto to download source notebook

python
jupyter
Quarto is one of the best platforms to create statically generated blogs with Markdown and Jupyter Notebooks.
May 28, 2024

 

AI Code Assistant at NERSC

python
nersc
If you are used to Github Copilot on VS Code or on Google Colaboratory as I am, coding without an AI assistant is so slow!
May 28, 2024

Interactive 3D plot of a Planck map with Matplotlib

python
astrophysics
This is an update of my older tutorial
May 20, 2024

 

HEALPix in Megapixels

cosmology
healpy
How many megapixels is a HEALPix map?
May 20, 2024

 

Load data from Globus in the browser and plot with pyscript

python
Globus supports a Javascript SDK which allows to authenticate against Globus Auth and then access all the Globus services, for example accessing protected data or setting up…
May 16, 2024

 

Ubuntu 22.04 Minimal on Jetstream with GPU Support

kubernetes
jetstream2
Just notes on how to create a Ubuntu 22.04 image with GPU support
May 13, 2024

 

News item on SDSC website about Advanced Simons Observatory and PySM

python
pysm
cosmology
In collaboration with the SDSC Communications team I prepared a news item for the SDSC Website announcing my participation in the Advanced Simons Observatory project…
May 12, 2024

 

Italian classes in San Diego

italian
My wife Maura D’Andrea is Principal and Teacher of Italian school of San Diego.
May 11, 2024

Generate point source maps with pixell

cosmology
pysm
Testing the pixell sim_objects functionality to create maps of point sources pre-smoothed with a gaussian beam. The purpose is to include this functionality in PySM to be…
May 2, 2024

 

Setup a conda environment at NERSC

python
nersc
NERSC recommends to use the “Global Common” to store Conda environments, because it is optimized to store lots of small files. Consider that it is mounted read-only on…
May 2, 2024

 

Flatcar Container Linux on Jetstream

kubernetes
jetstream2
Virtual Machine images provided by the Jetstream team are fully fledged and therefore quite large.
Apr 30, 2024

 

Automate sending polls via WhatsApp

Polls in WhatsApp are so convenient for the users, but so painful for group admins. Cannot modify them, cannot re-use or replicate them.
Apr 25, 2024

 

2-way Synchronization of Overleaf and Github via a Github Action

git
github
In the past I wrote tutorials on how to handle Github and Overleaf, the latest for example is this. However the Github button in Overleaf is fragile, it often gets…
Apr 17, 2024

Generate figures for PySM emissions

cosmology
healpy
pysm
This notebook generates some figures of Galactic and Extra-Galactic emissions using PySM. Mostly for displaying purposes.
Apr 12, 2024

 

Run PyTorch with GPU support on Expanse

hpc
jupyter
In this tutorial we will create a Python environment with PyTorch and see how to get Jupyterlab running on a GPU node.
Apr 4, 2024

 

Deploy Kubernetes on Jetstream 2 with GPU support

jetstream2
jupyterhub
This work has been supported by Indiana University and is cross-posted on the Jetstream 2 official documentation website.
Feb 8, 2024

 

Soft Scaling Kubernetes on Jetstream

kubernetes
jetstream2
ana-espinoza
Work and post contributed by Ana Espinoza, only tested and benchmarked by Andrea Zonca.
Feb 2, 2024

 

Make a PDF from a series of screenshots

bash
You might need to digitize some web content by taking screenshots and then bundling them up in a PDF.
Jan 12, 2024

 

Received DeSouza award for work on deploying software infrastructure on Jetstream

jupyterhub
jetstream
kubernetes
I am honored to have received the 2023 DeSouza award from Unidata, a NSF-funded center providing data, software and support for the field of Geoscience. The award celebrates…
Nov 2, 2023

 

Buy US Treasury bonds on Fidelity

The Fidelity interface is complicated, here a description of the steps necessary to buy newly issued US Treasury bonds at the next available auction. Currently auctions are…
Oct 27, 2023

 

Gateways 2023 tutorial about Dask and JupyterHub on Kubernetes on Jetstream

jupyterhub
jetstream
kubernetes
Video on Youtube
Oct 11, 2023

 

Deploy Dask Gateway with JupyterHub on Kubernetes

kubernetes
jetstream2
jupyterhub
dask
In this tutorial we will install Dask Gateway, currently version 2023.9.0, on Kubernetes and configure JupyterHub so Jupyter Notebook users can launch private Dask cluster…
Sep 28, 2023

 

Deploy Github Authenticator in JupyterHub

jupyterhub
jetstream
kubernetes
Updated June 2024: added more options to config file
Sep 27, 2023

 

Setup HTTPS on Kubernetes with cert-manager

kubernetes
jetstream
jupyterhub
Update March 2024: the routing issue that force cert-manager pods to run on the control-plane are back, see this Github issue, so we had to add back the pinning of…
Sep 26, 2023

 

Tutorial on Python for HPC at the SDSC Summer Institute

python
hpc
Each August the San Diego Supercomputer Center organizes a week-long summer school that teaches a large number of topics related to Supercomputing and AI to early career…
Aug 16, 2023

 

Deploy Kubernetes on Jetstream 2 with Kubespray 2.21.0

kubernetes
jetstream2
This work has been supported by Indiana University and is cross-posted on the Jetstream 2 official documentation website.
Jun 19, 2023

 

Hiring a Computational Scientist at the San Diego Supercomputer Center

hpc
Come work in my group of High Performance Computing “consultants” at the San Diego Supercomputer Center, help scientists develop, scale and run Data Intensive computations…
Apr 24, 2023

 

Cosmology webinar Hyper Supreme Cam

cosmology
Webinar about Cosmology results, in particular tension in \(S_8\) by the Hyper Supreme Cam collaboration:
Apr 23, 2023

 

Get a Letsencrypt certificate for Dreamhost with certbot

Dreamhost has discontinued offering free Letsecrypt certificated in their shared hosting.
Apr 1, 2023

 

Update Openstack credentials in Kubernetes

kubernetes
jetstream
If you are deploying Kubernetes on top of Openstack, the Openstack External cloud provider stores the ID and Secret necessary to authenticate with the cloud infrastructure…
Mar 23, 2023

Work on a Latex Document in Github Codespaces

git
github
I was exploring using Github Codespaces instead of Overleaf to work on a Latex document.
Mar 16, 2023

 

Gzipping or not Gzipping HEALPix maps

healpy
nersc
HEALPix maps are generally stored in FITS format, they can be Gzipped to save disk space. Reading gzipped maps is natively supported in healpy, however, it takes longer…
Mar 6, 2023

 

Share data from NERSC publicly via web

nersc
NERSC allows to make data which is on their Community File System publicly available via a web interface.
Feb 24, 2023

 

Showyourwork! Globus demo

python
Showyourwork! is a great project to make scientific articles reproducible.
Feb 21, 2023

 

Deploy a NFS server to share data between JupyterHub users on Jetstream

kubernetes
jetstream
jupyterhub
linux
This tutorial is a minor update of https://www.zonca.dev/posts/2020-07-10-nfs-server-kubernetes-jetstream.
Feb 6, 2023

 

Work on a Latex Document with Github and Overleaf

git
github
In the past I have written about coordinating a large document using multiple overleaf documents and Github repositories. In this post I will just layout the simpler case…
Feb 2, 2023

 

Deploy Kubernetes on Jetstream 2 with GPU support

jetstream2
jupyterhub
This tutorial is obsolete, see the new version of the tutorial.
Jan 23, 2023

Compare WebSky Radio Galaxies maps from PySM to Planck

healpy
pysm
I need to check that the amplitude of the maps of Radio Galaxies produced by WebSky are reasonable if compared to experimental results. The main worry is that being point…
Dec 5, 2022

 

Share data between JupyterHub users with Manila shares

jetstream2
kubernetes
This work has been supported by Indiana University and is cross-posted on the Jetstream 2 official documentation website.
Dec 5, 2022

Deploy BinderHub on top of Kubernetes on Jetstream 2

jetstream2
This work has been supported by Indiana University and is cross-posted on the Jetstream 2 official documentation website.
Nov 15, 2022

 

Setup a conda environment at NERSC

python
nersc
Obsolete: see the new version
Nov 8, 2022

 

Build and host Singularity containers on Github

singularity
I present a demo repository configured to build Singularity containers using Github actions and then host them using the Github container registry:
Nov 3, 2022

 

Generate Click command line options dynamically from class arguments

python
In this application I need to use command line options to create class objects. In order not to repeat all class arguments in the Click configuration, I created a simple…
Oct 26, 2022

 

Closures in Numba

python
Closures can be used to dinamically create different versions of a function based on some parameter, a similar functionality can be achieved with functools.partial.
Oct 24, 2022

 

Install the JupyROOT Python kernel in JupyterHub

jupyterhub
After installing ROOT’s conda package, for example with micromamba:
Oct 11, 2022

 

Singularity on Expanse tutorial

singularity
hpc
jetstream2
As one of my last tasks in XSEDE, I updated and improved the tutorial about running Singularity containers in the HPC system Espanse at my institution the San Diego…
Oct 5, 2022

Screenshot of old blog

Migrate from fastpages to quarto preserving git history

Most of my blog posts are in Markdown…however, there are cases where plotting and code are very important, and nothing beats a Jupyter Notebook for that.
Sep 27, 2022

 

Deploy MariaDB on Jetstream 2 on top of Kubernetes

kubernetes
jetstream2
In this tutorial we will install a MariaDB instance backed by a persistent volume on Jetstream 2. It will be in the jhub namespace, so that it can be accessed by the…
Jun 6, 2022

 

Remove unique cell id from Jupyter Notebooks

git
jupyter
I know! Jupyter is littering your git diff with randomly generated cell ids and nbstripout doesn’t remove them, (I’m sure they are useful for some reason).
May 17, 2022

 

Access running Github action with SSH

github
Sometimes Github actions are failing and it is difficult to reproduce the error locally, in particular if you have a different OS.
May 2, 2022

 

Monitor Restic backups on Kubernetes

jetstream2
kubernetes
For one of my production JupyterHub deployments on Kubernetes, I have setup an automated system to perform nightly backup of the user data, see the full tutorial on how to…
Apr 27, 2022

 

Jetstream2 SU calculator

jetstream2
Jupyter Notebook to compute daily, monthly, yearly consumption of SU based on the number and type of Virtual Machines:
Apr 15, 2022

 

Custos authentication for JupyterHub

kubernetes
jetstream2
jupyterhub
Custos is a security middleware used to authenticate users to Airavata-based Science Gateways. It is relevant to the Science Gateways community to unify authentication and…
Apr 13, 2022

 

Science Gateway with Dask and Zarr

kubernetes
jetstream2
jupyterhub
dask
This material was presented on April 2022 at the MiniGateways 2022 conference organized by the wonderful Science Gateways Community Institute (SGCI).
Apr 5, 2022

 

Use the distributed file format Zarr on Jetstream 2 object storage

dask
jetstream2
Zarr is a file format designed for cloud computing, see documentation.
Apr 4, 2022

 

Deploy Dask Gateway with JupyterHub on Kubernetes

kubernetes
jetstream2
jupyterhub
dask
Tutorial obsolete, see the new version of the tutorial
Apr 4, 2022

 

Deploy JupyterHub on Jetstream 2 on top of Kubernetes

kubernetes
jupyterhub
jetstream2
This tutorial is a followup to: Deploy Kubernetes on Jetstream 2 with Kubespray 2.18.0, so I’ll assume Kubernetes is already deployed with a default storageclass.
Mar 31, 2022

 

Deploy Kubernetes on Jetstream 2 with Kubespray 2.18.0

kubernetes
jetstream2
Obsolete: please use the updated release of this tutorial.
Mar 30, 2022

 

Workflow for Jupyter Notebooks under version control

jupyter
git
I’ll present here my strategy for keeping Jupyter Notebooks under version control.
Mar 22, 2022

 

Deploy Dask Gateway with JupyterHub on Kubernetes

kubernetes
jetstream
jupyterhub
dask
Tutorial OBSOLETE Please check the updated version of this tutorial.
Jan 27, 2022

 

Make for repeated commands

linux
I often find myself repeat the same commands in a project, for example in Openstack/Kubernetes, but also when I need to interface with Docker (build/push image).
Jan 7, 2022

 

Fund healpy via Github Sponsors

healpy
Github Sponsor page
Nov 8, 2021

 

Kubernetes certifications CKA and CKAD

kubernetes
I recently pursued 2 Kubernetes certifications by Linux Foundation:
Nov 4, 2021

Compare HEALPix and Gauss-Legendre pixelizations for sky emission modelling

healpy
cosmology
pysm
We compare the transform from spherical harmonics space to pixel space and back using HEALPix and Gauss-Legendre pixelization. The context is the evaluation of sky emission…
Nov 2, 2021

 

Deploy Hadoop on Kubernetes on Jetstream

kubernetes
jetstream
We are deploying the good old Hadoop on top of Kubernetes on Jetstream. Don’t ask why.
Oct 12, 2021

Plot HEALPix pixel boundaries

healpy
Some time ago I created a Jupyter Notebook based on plotly to create a 3D visualization of the boundaries of HEALPix pixels on the sphere.
Oct 6, 2021

 

Monitor traffic on Github repositories

github
Github only shows traffic data for the last 2 weeks. Through the API is possible to gather those data every 2 weeks and save them for later collection and reduction.
Oct 1, 2021

 

Manage Globus groups with the Python SDK

python
hpc
Globus is the best tool to transfer quickly Terabytes data between Supercomputers because it automatically parallelizes the transfer to saturate the network (yeah I know I…
Sep 24, 2021

 

New paper about PySM 3

openscience
python
pysm
cosmology
Update November 2021
Aug 4, 2021

 

Upload a JOSS paper to Arxiv

github
openscience
If you are in Astrophysics, you probably want to have your JOSS paper published to the Arxiv.
Aug 1, 2021

 

Configure dreamhost email

Configuration to read Dreamhost email from Gmail:
Jun 27, 2021

 

healpy 1.15.0 released

healpy
python
Just released healpy 1.15.0, this is a large release with many important changes, please test and open issues at https://github.com/healpy/healpy/issues
Jun 22, 2021

Investigate rotation of masks in healpy

healpy
cosmology
Investigating a question about healpy on Stackoverflow
Jun 18, 2021

 

Learn Italian in San Diego

italian
My wife Maura D’Andrea recently opened a new Italian language school in San Diego.
Jun 17, 2021

 

Migrate from Google Docs to Overleaf and Github

github
Once a document on Google Docs becomes too big and complicated to make sure it is consistent, it is a good idea to migrate it to Latex. In particular we can support both…
May 20, 2021

Correct a power spectrum by the instrument beam with healpy

healpy
cosmology
If you are analyzing a map from an instrument with a specific beam width, you can correct the power spectrum by the smoothing factor caused by that beam and obtain a better…
Apr 27, 2021

 

Backup Kubernetes volumes to OpenStorageNetwork object store

jetstream
kubernetes
In my specific scenario, I have users running JupyterHub on top of Kubernetes on the Jetstream XSEDE Cloud resouce. Each user has a persistent volume as their home folder of…
Apr 19, 2021

Science Gateways Tech blog - Kubernetes and JupyterHub on Jetstream

kubernetes
jetstream
This is cross-posted from the Science Gateways Community institute tech blog, thanks to Marlon Pierce and Katherine Lawrence for their feeback.
Mar 28, 2021

 

Stream video from object store on Jetstream

jetstream
In this tutorial we will serve video stored in Jetstream’s object store (like Amazon S3) as HTTP live streaming to a user’ browser. This makes it difficult for the user to…
Mar 25, 2021

Tutorial on how to rotate maps in healpy

In this notebook we will explore different ways of rotating maps in healpy.
Mar 11, 2021

Compute the Planck CMB temperature power spectrum with healpy anafast

healpy
cosmology
In this notebook we will load the Planck CMB-only temperature map and try to reproduce the Planck CMB power spectrum in temperature just using healpy.
Feb 28, 2021

 

Coordinate a large Latex document with multiple Overleaf projects and Github

git
github
In case we need to build a large document (~hundreds of pages) and we have a large number of collaborators (more than 50), it is convenient to have each section of the…
Jan 28, 2021

 

Autoscaling script for JupyterHub on top of Kubespray

kubernetes
jetstream
I think the most reliable way of deploying Kubernetes on Openstack is Kubespray, see my latest tutorial about it focused on Jetstream.
Jan 20, 2021

 

Deploy Kubernetes on Jetstream with Kubespray 2.15.0

kubernetes
jetstream
This is an update to previous tutorials, focused on deploying Kubernetes 1.19.7 (released in Jan 2021, based on 1.19.0 released in August 2020), compared to 1.17.6 of the…
Jan 20, 2021

 

CFITSIO file writing with checkpointing

astrophysics
I wrote a prototype implementation of writing FITS files in C checkpointing using the FITS flush function so that if the code segfaults, the file is not corrupted, see:
Nov 30, 2020

 

Paper review workflow with Overleaf, git and Google Docs

openscience
git
In this blog post I propose a workflow to use the collaborative features of Google Docs to review a paper being written in Overleaf (unless you have Overleaf Pro which has…
Nov 18, 2020

 

Install HEALPix and PolSpice in a conda environment

healpy
hpc
nersc
Some notes on how to install HEALPix and PolSpice inside a conda environment, with some details about doing it at NERSC, but most of the tutorial is independent of that.
Oct 14, 2020

Example of the healpy query disc function

python
healpy
cosmology
lonlat=True switches ang2vec from requiring colatitude \(\theta\) and longitude \(\phi\) in radians to longitude and latitude in degrees (notice that also the order changes)
Oct 2, 2020

 

Read and process Planck CMB power spectra with healpy

healpy
cosmology
In this tutorial we will load CMB power spectra generated by the cosmological parameters measured by Planck, we will create a realization of the Spherical Harmonics…
Sep 30, 2020

 

Edit video recordings with ffmpeg

A few scripts to speedup, concatenate and possibly other operations using ffmpeg on linux (also works on Chromebooks) from the command line:
Sep 25, 2020

 

How to share Jupyter Notebooks

jupyter
github
I tend to preserve and post online all the work I do for later reference and sharing. Jupyter Notebooks are a bit more complex than standard source code because they are…
Sep 15, 2020

 

Simple WCS with astropy modeling and gwcs

astrophysics
Some notes on how to convert a FITS WCS to a WCS from gwcs (Generalized WCS) to be used within the JWST pipeline
Sep 10, 2020

 

Gateways 2020 paper about Kubernetes and JupyterHub on Jetstream

jupyterhub
jetstream
kubernetes
My paper in collaboration with Richard Signell (USGS), Julien Chastang (UCAR), John Michael Lowe (IU/Jetstream), Jeremy Fischer (IU/Jetstream) and Robert Sinkovits (SDSC)…
Sep 4, 2020

Unit conversion with broadband detectors looking at the CMB

healpy
cosmology
We measure the sky with out broadband instrument, we assume we only measure the CMB solar dipole, initially the units are arbitrary, for example Volts of our instrument.
Sep 4, 2020

Unit conversion with broadband detectors looking at foregrounds

cosmology
We measure the sky with out broadband instrument, we assume we only measure the CMB solar dipole, initially the units are arbitrary, for example Volts of our instrument.
Sep 4, 2020

 

Migrate from Travis-CI and Readthedocs to Github actions

python
github
For a number of years I have been concerned about the duplication of work having to maintain continuous integration environments both on Travis-CI to run the unit tests and…
Sep 1, 2020

 

Redirect readthedocs documentation to another website

python
This is useful if you switch to hosting your own documentation, for example using Sphinx Multiversion on Github pages, tutorial coming soon.
Sep 1, 2020

 

Renew letsencrypt certificate with NGINX

linux
If you have a default domain already configured with NGINX, the easiest way to renew manually a domain is to temporarily disable your custom domain and then use certbot-auto …
Aug 24, 2020

 

Investigate broken implementation of bandpass unit conversion in PySM 3

cosmology
pysm
Investigation of the broken implementation of the bandpass unit conversion function in PySM 3 and check of its impact on existing Simons Observatory and CMB-S4 simulated…
Aug 24, 2020

 

Deploy Dask Gateway with JupyterHub on Kubernetes

kubernetes
cloudcomputing
jetstream
jupyterhub
dask
This tutorial is obsolete, please follow https://zonca.dev/2022/01/dask-gateway-jupyterhub.html
Aug 11, 2020

 

Dask array rounding

python
dask
Dask array rounding float32
Aug 4, 2020

 

Deploy a NFS server to share data between JupyterHub users on Jetstream

kubernetes
jetstream
jupyterhub
linux
Important: This tutorial is obsolete, please refer to the new version from February 2023
Jul 10, 2020

Handle white noise with healpy 3 not-uniform and partial sky coverage

cosmology
python
healpy
Simulate white noise maps and use hitmaps
Jun 21, 2020

Handle white noise with healpy 2 partial sky coverage

cosmology
python
healpy
Simulate white noise maps and use hitmaps
Jun 20, 2020

Handle white noise with healpy 1 Full sky coverage

cosmology
python
healpy
Simulate white noise maps and use hitmaps
Jun 19, 2020

 

Deploy Kubernetes on Jetstream with Kubespray

kubernetes
jetstream
This tutorial is obsolete, check the updated version of the tutorial
Jun 15, 2020

 

My own git cheatsheet

git
I noticed I always google the same procedures…
Jun 1, 2020

 

Import and export list of blocked users on Twitter

python
Back in 2015, Twitter provided the feature to export and import lists of blocked users, unfortunately they discontinued this service.
Jun 1, 2020

 

Minimal example of readthedocs configuration for conda

python
Prompted by the announced better support of conda in readthedocs, more memory! I setup a Python package with conda to automatically build the documentation on readthedocs.
May 27, 2020

 

Deploy Kubernetes and JupyterHub on Jetstream with Magnum

kubernetes
jetstream
jupyterhub
This tutorial deploys Kubernetes on Jetstream with Magnum and then JupyterHub on top of that using zero-to-jupyterhub.
May 21, 2020

White noise NET in Radio-astronomy and Cosmology

cosmology
python
healpy
Create a white noise map and compare with power spectrum expected from the NET
Apr 24, 2020

 

ECSS Symposium introduction to cloud computing with Jetstream and deployment of Kubernetes

jupyterhub
jetstream
kubernetes
ECSS Symposium, April 2020, Web presentation to the XSEDE Extended Collaborative Support Services.
Apr 20, 2020

 

Use the Jetstream object store

jetstream
I plan to collect here notes about using the Openstack object store on Jetstream.
Apr 1, 2020

 

Raise and check a flag array with numpy

python
Handle flag array bits with numpy bitwise operations
Mar 16, 2020

 

Setup HTTPS on Kubernetes with Letsencrypt

kubernetes
jetstream
jupyterhub
This tutorial is obsolete since September 2023, see the updated tutorial.
Mar 13, 2020

 

Kill Jupyter Notebook servers

python
jupyter
Just today I learned how to properly stop previously running Jupyter Notebook servers, here for future reference:
Mar 12, 2020

Migrate from Pelican to Fastpages

python
I have been using the Pelican static website generator for a few years, hosting the content on Github, automatically build on push via Travis-CI and deploy on Github pages to …
Mar 9, 2020

 

Deploy CVMFS on Kubernetes

kubernetes
jetstream
jupyterhub
CVMFS is a software distribution service, it is used by High Energy Physics experiments at CERN to synchronize software environments across the whole collaborations.
Feb 26, 2020

screenshot of add to Google Calendar

Organize calendars for a large scientific collaboration

openscience
Many scientific collaborations have a central calendar, often hosted on Google Calendar, to coordinate Teleconferences, meetings and events across timezones.
Dec 2, 2019

 

Simulate users on JupyterHub

kubernetes
jetstream
jupyterhub
Updated January 2021
Oct 30, 2019

 

Execute Jupyter Notebooks not interactively

jupyter
Over the years, I have explored how to scale up easily computation through Jupyter Notebooks by executing them not-interactively, possibily parametrized and remotely. This…
Sep 23, 2019

 

Deploy Cluster Autoscaler for Kubernetes on Jetstream

kubernetes
jetstream
jupyterhub
The Kubernetes Cluster Autoscaler is a service that runs within a Kubernetes cluster and when there are not enough resources to accomodate the pods that are queued to run…
Sep 12, 2019

 

Create a Github account for your research group with free private repositories

github
git
openscience
Github allows a research group to create their own webpage where they can host, share and develop their software using the git version control system and the powerful Github…
Aug 24, 2019

 

Ship large files with Python packages

python
It is often useful to ship large data files together with a Python package, a couple of scenarios are:
Aug 21, 2019

 

Deploy Kubernetes and JupyterHub on Jetstream with Magnum

kubernetes
jetstream
jupyterhub
This tutorial is obsolete, please checkout the updated version at https://zonca.dev/2020/05/kubernetes-jupyterhub-jetstream-magnum.html.
Jun 14, 2019

 

Webinar about distributed computing with Python

python
hpc
Recording available of the webinar I gave about “Distributed computing with Python”:
May 30, 2019

Kubernetes monitoring with Prometheus and Grafana

kubernetes
jetstream
jupyterhub
See the updated version of this tutorial
Apr 20, 2019

 

Inherit group permission in folder

hpc
linux
I have googled this so many times…
Mar 24, 2019

 

Kubernetes monitoring with Dashboard, Prometheus, and Grafana

kubernetes
jetstream
jupyterhub
Install with helm:
Feb 22, 2019

 

Scale Kubernetes manually on Jetstream

kubernetes
jetstream
jupyterhub
We would like to modify the number of Openstack virtual machines available to Kubernetes. Ideally we would like to do this automatically based on the load on JupyterHub…
Feb 22, 2019

 

Deploy Kubernetes with Kubespray 2.8.2 and JupyterHub with helm recipe 0.8 on Jetstream

kubernetes
jetstream
This tutorial is obsolete, check the updated version of the tutorial
Feb 22, 2019

 

Use the distributed file format Zarr on Jetstream Swift object storage, 2019

jupyterhub
jetstream
dask
This is an updated version of the 2018 edition
Jan 24, 2019

Deploy Pangeo on Kubernetes deployment on Jetstream created with Kubespray

kubernetes
jetstream
jupyterhub
The Pangeo collaboration for Big Data Geoscience maintains a helm chart with a prefigured JupyterHub deployment on Kubernetes which also supports launching private dask…
Dec 20, 2018

 

Setup two factor authentication for UCSD, and Lastpass

Starting at the end of January 2019 UCSD requires every employee to have activated two factor authentication.
Dec 12, 2018

 

Deploy JupyterHub on a Supercomputer for a workshop or tutorial 2018 edition

jupyterhub
hpc
I described how to deploy JupyterHub with each user session running on a different node of a Supercomputer in my paper for PEARC18, however things are moving fast in the…
Nov 7, 2018

 

Deploy JupyterHub on a Supercomputer for a workshop or tutorial 2018 edition

jupyterhub
hpc
I described how to deploy JupyterHub with each user session running on a different node of a Supercomputer in my paper for PEARC18, however things are moving fast in the…
Nov 7, 2018

 

Advanced pandas with Astrophysics example Notebook

jupyter
astrophysics
Taught a lesson today on advanced python and pandas based on an example application in Astrophysics with simulations of data from the Planck Satellite, features also a…
Oct 26, 2018

 

Bring your computing to the San Diego Supercomputer Center

hpc
I am often asked what computing resources are available at the San Diego Supercomputer Center for scientists and what is the best way to be granted access. I decided to…
Oct 24, 2018

 

Deploy JupyterHub on Kubernetes deployment on Jetstream created with Kubespray 3/3

kubernetes
jetstream
jupyterhub
All of the following assumes you are logged in to the master node of the Kubernetes cluster deployed with kubespray and checked out the repository:
Sep 24, 2018

 

Explore a Kubernetes deployment on Jetstream with Kubespray 2/3

kubernetes
jetstream
This is the second part of the tutorial on deploying Kubernetes with kubespray and JupyterHub on Jetstream.
Sep 23, 2018

 

Deploy Kubernetes on Jetstream with Kubespray 1/3

kubernetes
jetstream
This tutorial is obsolete, check the updated version of the tutorial
Sep 23, 2018

 

PEARC18 Paper on Deploying Jupyterhub at scale on XSEDE

singularity
jetstream
jupyterhub
kubernetes
Bob Sinkovits and I are presenting a paper at PEARC18 about:
Jul 23, 2018

 

Updated Singularity images for Comet

singularity
hpc
Back in January 2017 I wrote a blog post about running Singularity on Comet.
Jul 22, 2018

 

Create DockerHub auto build

github
It is very convenient to create Autobuild repositories on DockerHub linked to a Github repository with a Dockerfile. Then every time you commit to Github, Dockerhub is going…
Jul 19, 2018

 

How to organize code and data for simulations at NERSC

hpc
python
nersc
I recently improved my strategy for organizing code and data for simulations run at NERSC, I’ll write it here for reference.
Jun 20, 2018

 

Setup private dask clusters in Kubernetes alongside JupyterHub on Jetstream

jupyterhub
kubernetes
jetstream
dask
In this post we will leverage software made available by the Pangeo community to allow each user of a Jupyterhub instance deployed on Jetstream on top of Kubernetes to…
Jun 7, 2018

 

How to post a PEARC18 paper pre-print to Arxiv

openscience
Follows the step-by-step version:
May 12, 2018

 

Launch a shared dask cluster in Kubernetes alongside JupyterHub on Jetstream

jupyterhub
jetstream
dask
Let’s assume we have already a Kubernetes deployment and have installed JupyterHub, see for example my previous tutorial on Jetstream. Now that users can login and access a…
May 4, 2018

 

Install a BOINC server on Jetstream

jetstream
cloudcomputing
BOINC is the leading platform for volunteer computing.
Mar 29, 2018

 

Use the distributed file format Zarr on Jetstream Swift object storage

jupyter
jetstream
cloudcomputing
Updated again in January 2019
Mar 3, 2018

 

Use the distributed file format Zarr on Jetstream Swift object storage

jupyter
jetstream
Zarr is a pretty new file format designed for cloud computing, see documentation and a webinar for more details.
Mar 3, 2018

 

Install custom Python environment on Jupyter Notebooks at NERSC

jupyterhub
python
hpc
nersc
NERSC has provided a JupyterHub instance for quite some time to all NERSC users. It is currently running on a dedicated large-memory node on Cori, so now it can access also…
Dec 21, 2017

 

ECSS Symposium about Jupyterhub deployments on XSEDE

jupyterhub
jetstream
cloudcomputing
ECSS Symposium, 19 December 2017, Web presentation to the XSEDE Extended Collaborative Support Services.
Dec 15, 2017

 

Deploy scalable Jupyterhub with Kubernetes on Jetstream

jupyterhub
jetstream
cloudcomputing
The best infrastructure available to deploy Jupyterhub at scale is Kubernetes. Kubernetes provides a fault-tolerant system to deploy, manage and scale containers. The…
Dec 5, 2017

 

Store a conda environment inside a Notebook

jupyter
python
Last August, during the Container Analysis Environments Workshop held at Urbana-Champaign, we had discussion about reproducibility in the Jupyter Notebooks. There came out…
Dec 4, 2017

 

How to modify Singularity images on a Supercomputer

singularity
hpc
Singularity allows to run your own OS within most Supercomputers, see my previous post about Running Ubuntu on Comet via Singularity
Nov 6, 2017

 

Deploy scalable Jupyterhub on Docker Swarm mode

jupyterhub
jetstream
cloudcomputing
Jupyterhub genrally requires roughly 500MB per user for light data processing and many GB for heavy data processing, therefore it is often necessary to deploy it across…
Oct 26, 2017

 

Setup automated testing on a Github repository with Travis-ci

github
git
Start from the CUDA 8 image from NVIDIA:
Oct 25, 2017

 

Setup automated testing on a Github repository with Travis-ci

github
git
It is good practice in software development to implement extensive testing of the codebase in order to catch quickly any bug introduced into the code when implementing new…
Sep 6, 2017

 

Deployment of Jupyterhub with Globus Auth to spawn Notebook on Comet in Singularity containers

jupyterhub
singularity
hpc
Follow the instructions at https://github.com/zonca/singularity-comet to build images from the ubuntu_anaconda_jupyterhub.def and centos_anaconda_jupyterhub.def definition…
Aug 11, 2017

 

How to create pull requests on Github

git
github
Pull Requests are the web-based version of sending software patches via email to code maintainers. They allow a person that has no access to a code repository to submit a…
Jun 30, 2017

 

How to create pull requests on Github

git
github
cloudcomputing
Pull Requests are the web-based version of sending software patches via email to code maintainers. They allow athat has no access to a code repository to submit a code…
Jun 30, 2017

 

Deploy Jupyterhub on a Supercomputer with SSH Authentication

python
jupyterhub
hpc
The best way to deploy Jupyterhub with an interface to a Supercomputer is through the use of batchspawner. I have a sample deployment explained in an older blog post: https:…
May 16, 2017

 

Configure Globus on your local machine for GridFTP with XSEDE authentication

linux
All the commands are executed on your local machine, the purpose of this tutorial is to be able to use globus-url-copy to copy efficiently data back and forth between your…
Apr 19, 2017

 

Sample deployment of Jupyterhub in HPC on SDSC Comet

python
jupyterhub
hpc
I have deployed an experimental Jupyterhub service (ask me privately if you would like access) installed on a SDSC Cloud virtual machine that spawns single user Jupyter…
Feb 26, 2017

 

Customize your Python environment in Jupyterhub

python
jupyterhub
Usecase: You have access to a Jupyterhub server and you would like to install some packages but cannot use pip install and modify the systemwide Python installation.
Feb 24, 2017

 

Automated deployment of Jupyterhub with Ansible

jupyterhub
cloudcomputing
Last year I wrote some tutorials on simple deployments of Jupyterhub on Ubuntu 16.04 on the OpenStack deployment SDSC Cloud, even if most of the steps would also be suitable…
Feb 3, 2017

 

How to publish your research software to Github

git
github
openscience
Nowadays many journals require that the software used to produce results described in a scientific paper be made available publicly for other peers to be able to reproduce…
Feb 1, 2017

 

Run Ubuntu in HPC with Singularity

singularity
hpc
If your answer to any of those question is yes, read on! Otherwise, well, still read on, it’s awesome!
Jan 13, 2017

 

Jupyterhub Docker Spawner with GPU support

python
jupyterhub
Docker Spawner allows users of Jupyterhub to run Jupyter Notebook inside isolated Docker Containers. Access to the host NVIDIA GPU was not allowed until NVIDIA release the NV…
Oct 12, 2016

 

Jupyterhub deployment on multiple nodes with Docker Swarm

jupyterhub
cloudcomputing
This post is part of a series on deploying Jupyterhub on OpenStack tailored at workshops, in the previous posts I showed:
May 24, 2016

 

Quick Jupyterhub deployment for workshops with pre-built image

python
jupyterhub
This tutorial explains how to use a OpenStack image I already built to quickly deploy a Jupyterhub Virtual Machine that can provide a good initial setup for a workshop…
Apr 28, 2016

 

Deploy Jupyterhub on a Virtual Machine for a Workshop

python
jupyterhub
This tutorial describes the steps to install a Jupyterhub instance on a single machine suitable for hosting a workshop, suitable for having people login with training…
Apr 16, 2016

 

Use your own Python installation (kernel) in Jupyterhub

python
jupyterhub
Updated February 2017
Oct 5, 2015

IPython/Jupyter notebook setup on NERSC Edison

jupyter
python
hpc
nersc
This tutorial explains the setup to run an IPython Notebook on a computing node on the supercomputer Edison at NERSC and forward its port encrypted with SSH to the browser…
Sep 24, 2015

 

IPython/Jupyter notebook setup on SDSC Comet

python
hpc
jupyter
This tutorial explains the setup to run an IPython Notebook on a computing node on the supercomputer Comet at the San Diego Supercomputer Center and forward the port…
Sep 17, 2015

Run Jupyterhub on a Supercomputer

python
jupyterhub
hpc
Summary: I developed a plugin for Jupyterhub: RemoteSpawner, it has a proof-of-concept interface with the Supercomputer Gordon at UC San Diego to spawn IPython Notebook…
Apr 2, 2015

 

Accelerate groupby operation on pixels with Numba

python
astrophysics
Download the original IPython notebook
Mar 24, 2015

Screenshot Chromebook click for full resolution

Software Carpentry setup for Chromebook

linux
python
In this post I’ll provide instructions on how to install the main requirements of a Software Carpentry workshop on a Chromebook. Bash, git, IPython notebook and R.
Feb 10, 2015

 

Zero based indexing

python
For Europeans zero based indexing feels reasonable if we think of floors in a house, the lowest floor is ground floor, then 1st floor and so on.
Oct 22, 2014

 

Write unit tests as cells of IPython notebooks

python
jupyter
Plugin for py.test to write unit tests as cells in IPython notebooks:
Sep 30, 2014

 

How to perform code review for scientific software

github
git
openscience
Code review is the formal process where a programmer inspects in detail a piece of software developed by somebody else in order to improve code quality by catching bugs…
Aug 28, 2014

 

Create a Github account for your research group with free private repositories

github
git
openscience
See the updated version at https://zonca.github.io/2019/08/github-for-research-groups.html
Aug 19, 2014

 

Thoughts on a career as a computational scientist

hpc
Recently I’ve been asked what are the prospects of a wannabe computational scientist, both in terms of training and in terms of job opportunities.
Jun 5, 2014

 

Machine learning at scale with Python

python
hpc
My talk for the San Diego Data Science meetup: http://www.meetup.com/San-Diego-Data-Science-R-Users-Group/events/170967362/
Mar 20, 2014

 

Python on Gordon

hpc
python
Gordon has already a python environment setup which can be activated by loading the python module:
Mar 20, 2014

 

Build Software Carpentry lessons with Pelican

python
openscience
Software Carpentry offers bootcamps for scientist to teach basic programming skills. All the material, mainly about bash, git, Python and R is available on Github under…
Feb 26, 2014

 

openproceedings - Github/FigShare based publishing platform for conference proceedings

python
openscience
Github provides a great interface for gathering, peer reviewing and accepting papers for conference proceedings, the second step is to publish them on a website either in…
Feb 13, 2014

 

wget file from google drive

linux
Sometimes it is useful, even more if you have a chromebook, to upload a file to Google Drive and then use wget to retrieve it from a server remotely.
Jan 31, 2014

IPython notebook on Gordon

Run IPython Notebook on a HPC Cluster via PBS

python
hpc
The IPython notebook is a great tool for data exploration and visualization. It is suitable in particular for analyzing a large amount of data remotely on a computing node…
Dec 18, 2013

 

Joining San Diego Supercomputer Center

TL;DR Left UCSB after 4 years, got staff position at San Diego Supercomputer Center within UCSD, will be helping research groups analyze their data on Gordon and more. Still…
Dec 10, 2013

 

Published paper on Destriping Cosmic Microwave Background Polarimeter data

python
cosmology
TL;DR version:
Nov 20, 2013

Jiffylab Login page

Jiffylab multiuser IPython notebooks

python
jiffylab is a very interesting project by Preston Holmes to provide sandboxed IPython notebooks instances on a server using docker. There are several user cases, for example:
Oct 14, 2013

 

How to log exceptions in Python

python
Sometimes it is useful to just catch any exception, write details to a log file and continue execution.
Oct 1, 2013

 

Google Plus comments plugin for Pelican

python
github
There has been recently several discussions about whether comments are any useful on blogs I think it is important to find better ways to connect blogs to social networks.…
Sep 27, 2013

 

How to automatically build your Pelican blog and publish it to Github Pages

python
git
Something I like a lot about Jekyll, the Github static blog generator, is that you just push commits to your repository and Github takes care of re-building and publishing…
Sep 26, 2013

 

clviewer, interactive plot of CMB spectra

python
astrophysics
Today it was HackDay at .Astronomy, so I felt compelled to hack something around myself, creating something I have been thinking for a while after my previous work on Interac…
Sep 17, 2013

Planck CMB map at high resolution

python
astrophysics
healpy
Prompted by a colleague, I created a high-resolution version of the Cosmic Microwave Background map in MollWeide projection released by the Planck collaboration, available…
Sep 10, 2013

Run Hadoop Python jobs on Amazon with MrJob

git
python
MrJob makes the implementation of the reducer particularly simple. Using hadoop-streaming directly, we needed also to first parse back the output of the mapper into python…
Sep 2, 2013

 

Interactive figures in the browser - CMB Power Spectra

astrophysics
healpy
For a long time I’ve been curious about trying out d3.js , the javascript plotting library which is becoming the standard for interactive plotting in the browser.
Aug 30, 2013

 

Planck CTP angular power spectrum ell binning

healpy
Planck released a binning of the angular power spectrum in the Explanatory supplement,
unfortunately the file is in PDF format, non easily machine-readable:

http://www.s…
Aug 20, 2013

HEALPix map of the Earth using healpy

healpy
HEALPix maps can also be used to create equal-area pixelized maps of the Earth, RGB colors are not supported in healpy, so we need to convert the image to colorscale.
The…
Aug 8, 2013

 

Export google analytics data via API with Python

python
Fun weekend hacking project: export google analytics data using the google APIs.

Clone the latest version of the API client from:

https://code.google.com/p/google-api-p…
Aug 4, 2013

Processing sources in Planck maps with Hadoop and Python

hpc
python
The purpose of this post is to investigate how to process in parallel sources extracted from full sky maps, in this case the maps release by Planck, using Hadoop instead of…
Jul 15, 2013

 

How to use the IPython notebook on a small computing cluster

hpc
python
The IPython notebook is a powerful and easy to use interface for using Python and particularly useful when running remotely, because it allows the interface to run locally…
Jun 22, 2013

 

IPython parallell setup on Carver at NERSC

hpc
python
IPython parallel is one of the easiest ways to spawn several Python sessions on a Supercomputing cluster and process jobs in parallel.

On Carver, the basic setup is…
Apr 11, 2013

 

Simple Mixin usage in python

python
One situation where Mixins are useful in Python is when you need to modify  a method of similar classes that you are importing from a package.
Apr 8, 2013

 

Noise in spectra and map domain

healpy
astrophysics
\(C_\ell\) of the variance map is just the variance map multiplied by the pixel area divided by the integration time.

\[C_\ell = \Omega_{\rm pix} \langle…
Apr 8, 2013

 

Basic fork/pull git workflow

git
Typical simple workflow for a (github) repository with few users.
Apr 6, 2013

Interactive 3D plot of a sky map

python
astrophysics
update May 2024: See the new version of this tutorial
Mar 12, 2013

 

How to cite HDF5 in bibtex

here the bibtex entry:


reference:
http://www.hdfgroup.org/HDF5-FAQ.html#gcite
Feb 27, 2013

 

Compile healpix C++ to javascript

linux
Compile C++ -> LLVM with clang

Convert LLVM -> Javascript:
https://github.com/kripken/emscripten/wiki/Tutorial
Jan 28, 2013

 

Elliptic beams, FWHM and ellipticity

astrophysics
The relationship between the Full Width Half Max, FWHM (min, max, and average) and the
ellipticity is:

FWHM = sqrt(FWHM_min * FWHM_max)
e = FWHM_max/FWHM_min

Jan 18, 2013

 

Ubuntu PPA for HEALPix and healpy

healpy
linux

Dec 17, 2012

 

Butterworth filter with Python

python
Using IPython notebook of course:

http://nbviewer.ipython.org/3843014/
Oct 6, 2012

 

IPython.parallel for Planck data analysis at NERSC

python
Planck is a Space mission for high precision measurements of the Cosmic Microwave Background (CMB), data are received as timestreams of output voltages from the 2…
Sep 27, 2012

 

homepage on about.me

moved my homepage to about.me:

http://about.me/andreazonca

it is quite nice, and essential, as most of it is just links to other websites, i.e. arXiv for…
Sep 26, 2012

 

doctests and unittests happiness 2

python
nosetests -v –with-doctest
Doctest: healpy.pixelfunc.ang2pix … ok
Doctest: healpy.pixelfunc.get_all_neighbours … ok
Doctest: healpy.pixelfunc.get_interp_val … ok
Doctest…
Aug 16, 2012

 

compile python module with mpi support

CC=mpicc LDSHARED=“mpicc -shared” python setup.py build_ext -i
Jul 6, 2012

 

some python resources

python
python tutorial:
http://docs.python.org/tutorial/
numpy tutorial [arrays]:
http://www.scipy.org/Tentative_NumPy_Tutorial
plotting tutorial:
http://matplotlib.sourc…
Nov 1, 2011

 

cfitsio wrapper in python

python
After several issues with pyfits, and tired of it being so overengineered, I’ve wrote my own fits I/O package in python, wrapping the C library cfitsio with ctypes.

Pretty…
Jun 21, 2011

 

unit testing happiness

python
nosetests -v
test_all_cols (pycfitsio.test.TestPyCfitsIoRead) ... ok
test_colnames (pycfitsio.test.TestPyCfitsIoRead) ... ok
test_move (pycfitsio.test.TestPyCfitsIoRead) ... ok
t…
Jun 21, 2011

Pink noise (1/f noise) simulations in numpy

python
https://gist.github.com/979729

May 18, 2011

 

Vim regular expressions

linux
very good reference of the usage of regular expressions in VIM:

http://www.softpanorama.org/Editors/Vimorama/vim_regular_expressions.shtml
Apr 29, 2011

 

set python logging level

python
often using logging.basicConfig is useless because if the logging module is already configured upfront by one of the imported libraries this is ignored.

The solution is…
Apr 13, 2011

 

pyfits memory leak in new_table

python
astrophysics
I found a memory leakage issue in pyfits.new_table, data were NOT deleted when the table was deleted, I prepared a test on github, using objgraph , which shows that data…
Mar 28, 2011

 

ipython and PyTrilinos

python

  • start ipcontroller

  • start ipengines:
    mpiexec -n 4 ipengine –mpi=pytrilinos

  • start ipython 0.11:
    import PyTrilinos
    from IPython.kernel import client
    mec =…





    •
    Feb 16, 2011

     

    git make local branch tracking origin

    git
    git branch –set-upstream master origin/master

    you obtain the same result as initial cloning
    Feb 2, 2011

     

    memory map npy files

    python
    Mem-map the stored array, and then access the second row directly from disk:

    X = np.load(‘/tmp/123.npy’, mmap_mode=‘r’)
    Jan 7, 2011

     

    force local install of python module

    python
    python setup.py install –prefix FOLDER


    creates lib/python2.6/site-packages, to force a local install you should use:

    python setup.py install –install-lib FOLDER
    Dec 3, 2010

     

    gnome alt f2 popup launcher

    linux
    
    gnome-panel-control –run-dialog
    Aug 31, 2010

     

    switch to interactive backend with ipython -pylab

    python
    objective:
    Aug 21, 2010

     

    numpy dtypes and fits keywords

    python
    bool: ‘L’,
    uint8: ‘B’,
    int16: ‘I’,
    int32: ‘J’,
    int64: ‘K’,
    float32: ‘E’,
    float64: ‘D’,
    complex64: ‘C’,
    complex128: ‘M’
    Aug 4, 2010

     

    count hits with numpy

    python
    I have an array where I record hits
    a=np.zeros(5)
    and an array with the indices of the hits, for example I have 2 hits on index 2
    hits=np.array([2,2])
    so I want to…
    Jul 23, 2010

     

    change column name in a fits with pyfits

    python
    no way to change it manipulating the dtype of the data array.
    a=pyfits.open(‘filename.fits’)
    a[1].header.update(‘TTYPE1’,‘newname’)
    you need to change the header…
    Jun 30, 2010

     

    healpix coordinates

    astrophysics
    Healpix considers latitude theta from 0 on north pole to pi south pole,
    so the conversion is:
    theta = pi/2 - latitude
    longitude and phi instead are consistently…
    Jun 23, 2010

     

    parallel computing the python way

    python
    forget MPI:
    http://showmedo.com/videotutorials/series?name=N49qyIFOh
    Jun 21, 2010

     

    quaternions for python

    python
    the situation is pretty problematic, I hope someday scipy will add a python package for rotating and interpolating quaternions, up to now:
    Jun 21, 2010

     

    change permission recursively to folders only

    linux
    find . -type d -exec chmod 777 {} ;
    Mar 23, 2010

     

    aptitude search ‘and’

    linux
    this is really something really annoying about aptitude, if you run:
    aptitude search linux headers
    it will make an ‘or’ search…to perform a ‘and’ search, which I need…
    Mar 16, 2010

     

    using numpy dtype with loadtxt

    linux
    python
    Let’s say you want to read a text file like this:


    Mar 3, 2010

     

    Stop ipcluster from a script

    python
    Ipcluster is easy to start but not trivial to stop from a script, after having finished the processing, here’s the solution:
    from IPython.kernel import client
    mec =…
    Feb 19, 2010

     

    Correlation

    Expectation value or first moment of a random variable is the probability weighted sum of the possible values (weighted mean).
    Expectation value of a 6-dice is…
    Jan 28, 2010

     

    execute bash script remotely with ssh

    linux
    a bash script launched remotely via ssh does not load the environment, if this is an issue it is necessary to specify –login when calling bash:

    ssh user@remoteserver.com…
    Jan 7, 2010

     

    lock pin hold a package using apt on ubuntu

    linux
    set hold:
    echo packagename hold | dpkg –set-selections

    check, should be hi :
    dpkg -l packagename

    unset hold:
    echo packagename install | dpkg –set-selections
    Jan 7, 2010

     

    load arrays from a text file with numpy

    python
    space separated text file with 5 arrays in columns:

    [sourcecode language=“python”]
    ods,rings,gains,offsets,rparams = np.loadtxt(filename,unpack=True)
    [/sourcecode]

    qui…
    Jan 5, 2010

     

    Latest Maxima and WxMaxima for Ubuntu Karmic

    linux
    http://zeus.nyf.hu/~blahota/maxima/karmic/

    on maxima mailing lists they suggested to install the sbcl built, so I first installed sbcl from the Ubuntu repositories and…
    Dec 15, 2009

     

    number of files in a folder and subfolders

    linux
    folders are not counted
    find . -type f | wc -l
    Dec 10, 2009

     

    forcefully unmount a disk partition

    linux
    check which processes are accessing a partition:

    [sourcecode language=“python”]lsof | grep ‘/opt’[/sourcecode]

    kill all the processes accessing the partition (check…
    Sep 17, 2008

     

    netcat, quickly send binaries through network

    linux
    just start nc in server mode on localhost:

    [sourcecode language=‘python’] nc -l -p 3333 [/sourcecode]

    send a string to localhost on port 3333:

    [sourcecode…
    Apr 29, 2008

     

    Decibels, dB and dBm, in terms of Power and Amplitude

    astrophysics
    It’s not difficult, just always having some doubts…
    Mar 29, 2008

     

    Relation between Power density and temperature in an antenna

    astrophysics
    Considering an antenna placed inside a blackbody enclosure at temperature T, the power received per unit bandwidth is:
    \(latex \omega = kT\)

    where k is Boltzmann…
    Mar 28, 2008

     

    Producing PDF from XML files

    linux
    I need to produce formatted pdf from XML data input file.
    The more standard way looks like to use XSL stylesheets.
    Associating a XSL sheet to an XML file permits most…
    Mar 28, 2008

     

    vim costumization

    linux
    it is about perl but it suggests very useful tricks for programming with vim
    http://mamchenkov.net/wordpress/2004/05/10/vim-for-perl-developers/
    Oct 17, 2006

     

    using gnu find

    linux
    list all the directories excluding “.”:
    Oct 3, 2006

     

    beginners bash guide

    linux
    great guide with many examples:

    http://tille.xalasys.com/training/bash/
    Oct 3, 2006

     

    tar quickref

    linux
    compress: tar cvzf foo.tgz .cc .h
    check inside: tar tzf foo.tgz | grep file.txt
    extract: tar xvzf foo.tgz
    extract 1 file only: tar xvzf foo.tgz path/to/file.txt
    Sep 25, 2006

     

    software carpentry

    linux
    basic software for scientists and engineers:
    http://www.swc.scipy.org/
    Sep 25, 2006

     

    Software libero per il trattamento di dati scientifici

    linux
    nella ricerca del miglior ambiente per analisi di dati scientifici da leggere questi articoli:

    http://www.pluto.it/files/journal/pj0501/swlibero-scie1.html

    http://www.pl…
    Sep 22, 2006

     

    command line processing

    linux
    Very useful summary of many linux command line processing tools (great perl onliners)

    http://grad.physics.sunysb.edu/~leckey/personal/forget/
    Sep 22, 2006

     

    awk made easy

    linux
    awk ’/REGEX/ {print NR “ $9” $4”_“\(5 ;}' file.txt </strong> <br/> supports extended REGEX like perl ( e.g. [:blank:] Space or tab characters ) <br/> NR is line…
    Sep 22, 2006

     

    pillole di astrofisica

    astrophysics
    curiosita’ ben spiegate da annibale d’ercole, interessante l’idea di avere un livello base e un livello avanzato
    http://www.bo.astro.it/sait/spigolature/spigostart.html
    Sep 20, 2006
    No matching items