Congratulations to the Debian Community
The Debian Project just released version 11 (aka “bullseye”) of their free operating system. In total, over 6,208 contributors worked on this release and were indispensable in making this launch happen. We would like to thank everyone involved for their combined efforts, hard work, and many hours pent in recent years building this new release that will benefit the entire open source community.
We would also like to acknowledge our in-house Debian developers who contributed to this effort. We really appreciate the work you do on behalf of the community and stand firmly behind your contributions.
What’s New in Debian 11 Bullseye
Debian 11 comes with a number of meaningful changes and enhancements. The new release includes over 13,370 new software packages, for a total of over 57,703 packages on release. Out of these, 35,532 packages have been updated to newer versions, including an update in the kernel from 4.19 in “buster” to 5.10 in bullseye.
Bullseye expands on the capabilities of driverless printing with Common Unix Printing System (CUPS) and driverless scanning with Scanner Access Now Easy (SANE). While it was possible to use CUPS for driverless printing with buster, bullseye comes with the package ipp-usb, which allows a USB device to be treated as a network device and thus extend driverless printing capabilities. SANE connects to this when set up correctly and connected to a USB port.
As in previous releases, Debian 11 comes with a Debian Edu / Skolelinux version. Debian Edu has been a complete solution for schools for many years. Debian Edu can provide the entire network for a school and then only users and machines need to be added after installation. This can also be easily managed via the web interface GOsa².
Debian 11 bullseye can be downloaded here.
For more information and greater technical detail on the new Debian 11 release, please refer to the official release notes on Debian.org
Contributions by Instaclustr Employees
Our Debian roots run deep here. credativ, which was acquired by Instaclustr in March 2021, has always been an active part of the Debian community and visited every DebConf since 2004. Debian also serves as the operating system at the heart of the Instaclustr Managed Platform.
For the release of Debian 11, our team has taken over various responsibilities in the community. Our contributions include:
- 90% of the PostgreSQL packaging of the new release Maintenance work on various packages
- Support as Debian-Sys-Admin
- Contributions to Debian Edu/Skolelinux
- Development work on kernel images
- Development work on cloud images
- Development work for various Debian backports
- Work on salsa.debian.org
Many of our colleagues have made significant contributions to the current release, including:
- Adrian Vondendriesch
- Alexander Wirt (Formorer)
- Bastian Blank (waldi)
- Christoph Berg (Myon)
- Dominik George (natureshadow)
- Felix Geyer (fgeyer)
- Martin Zobel-Helas (zobel)
- Michael Banck (azeem)
- Michael Meskes (feivel)
- Noël Köthe (Noel)
- Sven Bartscher (kritzefitz)
How to Upgrade
Given that Debian 11 bullseye is a major release, we suggest that everyone running on Debian 10 buster upgrade. The main steps for an upgrade include:
- Make sure to backup any data that should not get lost and prepare for recovery
- Remove non-Debian packages and clean up leftover files and old versions
- Upgrade to latest point release
- Check and prepare your APT source-list files by adding the relevant Internet sources or local mirrors
- Upgrade your packages and then upgrade your system
You can find a more detailed walkthrough of the upgrade process in the Debian documentation.
All existing credativ customers who are running a Debian-based installation are naturally covered by our service and support and are encouraged to reach out.
If you are interested in upgrading from your old Debian version, or if you have questions with regards to your Debian infrastructure, do not hesitate to drop us an email or contact us at firstname.lastname@example.org.
Or, you can get started in minutes with any one of these open source technologies like Apache Cassandra, Apache Kafka, Redis, and OpenSearch on the Instaclustr Managed Platform. Sign up for a free trial today.
100% Open Source – 100% Cost Control
The PostgreSQL® Competence Center of the credativ Group announces the creation of a new comprehensive service and support package that includes all services necessary for the operation of PostgreSQL® in enterprise environments. This new offering will be available starting August 1st, 2020.
“Motivated by the requirements of many of our customers, we put together a new and comprehensive PostgreSQL® service package that meets all the requirements for the operation of PostgreSQL® in enterprise environments.”, says Dr. Michael Meskes, managing director of the credativ Group.
“In particular, this package focuses on true open source support, placing great emphasis on the absence of any proprietary elements in our offer. Despite this, our service package still grants all of the necessary protection for operation in business-critical areas. Additionally, with this new offering, the number of databases operated within the company’s environment does not matter. As a result, credativ offers 100% cost control while allowing the entire database environment to be scaled as required.”
Database operation in enterprise environments places very high demands on the required service and support. Undoubtedly an extremely powerful, highly scalable, and rock-solid relational database is the basis for secure and high-performance operation.
However, a complete enterprise operating environment consists of much more than just the pure database; one needs holistic lifecycle management. Major and Minor version upgrades, migrations, security, services, patch management, and Long-Term Support (LTS) are just a few essential factors. Additionally, staying up to date also requires continuous regular training and practice.
Services for the entire operating environment
Beyond the database itself, one also needs a stable and highly scalable operating environment providing all necessary Open Source tools for PostgreSQL and meeting all requirements regarding high availability, security, performance, database monitoring, backups, and central orchestration of the entire database infrastructure. These tools include the open-source versions of numerous PostgreSQL related project packages, such as pgAdmin, pgBadger, pgBackrest, Patroni, but also the respective operating system environment and popular projects like Prometheus and Grafana, or even cloud infrastructures based on Kubernetes.
Just as indispensable as the accurate functioning of the database is smooth interaction with any components connected with the database. Therefore it is important to include and consider these components as well. Only when all aspects, such as operating system, load balancer, web server, application server, or PostgreSQL cluster solutions, work together, will the database achieve optimal performance.
This new support package is backed up by continuous 24×7 enterprise support, with guaranteed Service Level Agreements and all necessary services for the entire database environment, including a comprehensive set of open-source tools commonly used in today’s enterprise PostgreSQL environments. All of these requirements are covered by the PostgreSQL Enterprise package from credativ and are included within the scope of services. The new enterprise service proposal is offered at an annual flat rate, additionally simplifying costs and procurement.
The credativ Group is an independent consulting and services company with primary locations in Germany, the United States, and India.
Since 1999, credativ has focused entirely on the planning and implementation of professional business solutions using Open Source software. Since May 2006, credativ operates the Open Source Support Center (OSSC), offering professional 24×7 enterprise support for numerous Open Source projects.
In addition, our PostgreSQL Competence Center of credativ provides a dedicated database team a comprehensive service for the PostgreSQL object-relational database eco-system.
This article was originally written by Philip Haas.
Open Source Summit is the world’s largest, all-encompassing open source conference. Topics such as the latest infrastructure software, development on the Linux kernel and current works in open source communities are discussed. Until now, open source databases were a missing part of the conference program.
Together with Sunil Kamath (Microsoft) and Divya Bhargov (Pivotal), our Managing Director Dr. Michael Meskes forms the program committee for the new track of the event.
Dr. Michael Meskes comments this in his blog post at the Linux Foundation as follows:
“The open source database track will feature topics specific to databases themselves and their integration to the computing backbone for applications. The track will focus on databases of all kinds, as long as they are open source, and any deployment and integration topics.”
The complete blog post of the Linux Foundation can be found here.
This year’s Open Source Summit North America will take place in Austin, Texas. Whereas the Open Source Summit Europe will be hosted in Dublin, Ireland. Both events are once again backed by credativ with a sponsorship.
This article was originally written by Philip Haas.
The PostgreSQL® Global Development Group (PGDG) has released version 12 of the popular free PostgreSQL® database. As our article for Beta 4 has already indicated, a number of new features, improvements and optimizations have been incorporated into the release. These include among others:
Optimized disk space utilization and speed for btree indexes
btree-Indexes, the default index type in PostgreSQL®, has experienced some optimizations in PostgreSQL® 12.
btree indexes used to store duplicates (multiple entries with the same key values) in an unsorted order. This has resulted in suboptimal use of physical representation in these indexes. An optimization now stores these multiple key values in the same order as they are physically stored in the table. This improves disk space utilization and the effort required to manage corresponding btree type indexes. In addition, indexes with multiple indexed columns use an improved physical representation so that their storage utilization is also improved. To take advantage of this in PostgreSQL® 12, however, if they were upgraded to the new version using
pg_upgrade via a binary upgrade, these indexes must be recreated or re-indexed.
Insert operations in btree indexes are also accelerated by improved locking.
Improvements for pg_checksums
credativ has contributed an extension for pg_checksums that allows to enable or disable block checksums in stopped PostgreSQL® instances. Previously, this could only be done by recreating the physical data representation of the cluster using
pg_checksums now has the option to display a status history on the console with the
--progress parameter. The corresponding code contributions come from the colleagues Michael Banck and Bernd Helmle.
Optimizer Inlining of Common Table Expressions
Up to and including PostgreSQL® 11, the PostgreSQL® Optimizer was unable to optimize common table expressions (also called CTE or WITH queries). If such an expression was used in a query, the CTE was always evaluated and materialized first before the rest of the query was processed. This resulted in expensive execution plans for more complex CTE expressions. The following generic example illustrates this. A join is given with a CTE expression that filters all even numbers from a numeric column:
WITH t_cte AS (SELECT id FROM foo WHERE id % 2 = 0) SELECT COUNT(*) FROM t_cte JOIN bar USING(id);
In PostgreSQL® 11, using a CTE always leads to a CTE scan that materializes the CTE expression first:
EXPLAIN (ANALYZE, BUFFERS) WITH t_cte AS (SELECT id FROM foo WHERE id % 2 = 0) SELECT COUNT(*) FROM t_cte JOIN bar USING(id) ; QUERY PLAN ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── Aggregate (cost=2231.12..2231.14 rows=1 width=8) (actual time=48.684..48.684 rows=1 loops=1) Buffers: shared hit=488 CTE t_cte -> Seq Scan on foo (cost=0.00..1943.00 rows=500 width=4) (actual time=0.055..17.146 rows=50000 loops=1) Filter: ((id % 2) = 0) Rows Removed by Filter: 50000 Buffers: shared hit=443 -> Hash Join (cost=270.00..286.88 rows=500 width=0) (actual time=7.297..47.966 rows=5000 loops=1) Hash Cond: (t_cte.id = bar.id) Buffers: shared hit=488 -> CTE Scan on t_cte (cost=0.00..10.00 rows=500 width=4) (actual time=0.063..31.158 rows=50000 loops=1) Buffers: shared hit=443 -> Hash (cost=145.00..145.00 rows=10000 width=4) (actual time=7.191..7.192 rows=10000 loops=1) Buckets: 16384 Batches: 1 Memory Usage: 480kB Buffers: shared hit=45 -> Seq Scan on bar (cost=0.00..145.00 rows=10000 width=4) (actual time=0.029..3.031 rows=10000 loops=1) Buffers: shared hit=45 Planning Time: 0.832 ms Execution Time: 50.562 ms (19 rows)
This plan first materializes the
CTE with a
sequential scan with a corresponding filter
(id % 2 = 0). Here no functional index is used, therefore this scan is correspondingly more expensive. Then the result of the
CTE is linked to the table
Hash Join with the corresponding
Join condition. With PostgreSQL® 12, the optimizer now has the ability to inline these CTE expressions without prior materialization. The underlying optimized plan in PostgreSQL® 12 will look like this:
EXPLAIN (ANALYZE, BUFFERS) WITH t_cte AS (SELECT id FROM foo WHERE id % 2 = 0) SELECT COUNT(*) FROM t_cte JOIN bar USING(id) ; QUERY PLAN ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── Aggregate (cost=706.43..706.44 rows=1 width=8) (actual time=9.203..9.203 rows=1 loops=1) Buffers: shared hit=148 -> Merge Join (cost=0.71..706.30 rows=50 width=0) (actual time=0.099..8.771 rows=5000 loops=1) Merge Cond: (foo.id = bar.id) Buffers: shared hit=148 -> Index Only Scan using foo_id_idx on foo (cost=0.29..3550.29 rows=500 width=4) (actual time=0.053..3.490 rows=5001 loops=1) Filter: ((id % 2) = 0) Rows Removed by Filter: 5001 Heap Fetches: 10002 Buffers: shared hit=74 -> Index Only Scan using bar_id_idx on bar (cost=0.29..318.29 rows=10000 width=4) (actual time=0.038..3.186 rows=10000 loops=1) Heap Fetches: 10000 Buffers: shared hit=74 Planning Time: 0.646 ms Execution Time: 9.268 ms (15 rows)
The advantage of this method is that there is no initial materialization of the CTE expression. Instead, the query is executed directly with a
Join. This works for all non-recursive CTE expressions without side effects (for example, CTEs with write statements) and those that are referenced only once per query. The old behavior of the optimizer can be forced with the
WITH ... AS MATERIALIZED ... directive.
Generated Columns in PostgreSQL® 12 are materialized columns, which calculate a result based on expressions using existing column values. These are stored with the corresponding result values in the tuple. The advantage is that there is no need to create triggers for subsequent calculation of column values. The following simple example illustrates the new functionality using a price table with net and gross prices:
CREATE TABLE preise(netto numeric, brutto numeric GENERATED ALWAYS AS (netto * 1.19) STORED); INSERT INTO preise VALUES(17.30); INSERT INTO preise VALUES(225); INSERT INTO preise VALUES(247); INSERT INTO preise VALUES(19.15); SELECT * FROM preise; netto │ brutto ───────┼───────── 17.30 │ 20.5870 225 │ 267.75 247 │ 293.93 19.15 │ 22.7885 (4 rows)
brutto is calculated directly from the net price. The keyword
STORED is mandatory. Of course, indexes can also be created on
Generated Columns, but they cannot be part of a primary key. Furthermore, the SQL expression must be unique, i.e. it must return the same result even if the input quantity is the same. Columns declared as
Generated Columns cannot be used explicitly in
UPDATE operations. If a column list is absolutely necessary, the corresponding value can be indirectly referenced with the keyword
Omission of explicit OID columns
Explicit OID columns have historically been a way to create unique column values so that a table row can be uniquely identified database-wide. However, for a long time PostgreSQL® has only created these explicitly and considered their basic functionality obsolete. With PostgreSQL® the possibility to create such columns explicitly is now finally abolished. This means that it will no longer be possible to specify the
WITH OIDS directive for tables. System tables that have always referenced OID objects uniquely will now return OID values without explicitly specifying OID columns in the result set. Especially older software, which handled catalog queries carelessly, could get problems with a double column output.
Up to and including PostgreSQL® 11, database recovery and streaming replication instances were configured via a separate configuration file
With PostgreSQL® 12, all configuration work done there now migrates to
recovery.conf file is no longer required. PostgreSQL® 12 refuses to start as soon as this file exists. Whether recovery or streaming standby is desired is now decided either by a
recovery.signal file (for recovery) or by a
standby.signal file (for standby systems). The latter has priority if both files are present. The old parameter
standby_mode, which controlled this behavior since then, has been removed.
For automatic deployments of high-availability systems, this means a major change. However, it is now also possible to perform corresponding configuration work almost completely using the
ALTER SYSTEM command.
With PostgreSQL® 12 there is now a way to re-create indexes with as few locks as possible. This greatly simplifies one of the most common maintenance tasks in very write-intensive databases. Previously, a combination of
CREATE INDEX CONCURRENTLY and
DROP INDEX CONCURRENTLY had to be used. In doing so, it was also necessary to ensure that index names were reassigned accordingly, if required.
The release notes give an even more detailed overview of all new features and above all incompatibilities with previous PostgreSQL® versions.
Yesterday, the fourth beta of the upcoming PostgreSQL®-major version 12 was released.
Compared to its predecessor PostgreSQL® 11, there are many new features:
- Performance improvements for indexes: btree indexes now manage space more efficiently. The REINDEX command now also supports CONCURRENTLY, which was previously only possible with new indexes.
- WITH queries are now embedded in the main query and thus optimized much better by the planner. Previously, WITH queries were always executed independently.
- The native partitioning was further improved. Foreign keys can now also reference partitioned tables. Maintenance commands such as ATTACH PARTITION no longer require an exclusive table lock.
- The support of page checksums and the tool pg_checksums was further improved, also with substantial cooperation by credativ.
- It is now possible to integrate additional storage engines. The “zheap”, which is still under development, will be based on this, which promises more compact data storage with less bloat.
Of course, PostgreSQL® 12 will be tested using sqlsmith, the SQL “fuzzer” from our colleague Andreas Seltenreich. Numerous bugs in different PostgreSQL® versions were found with sqlsmith by using randomly generated SQL queries.
Debian and Ubuntu packages for PostgreSQL® 12 are going to be published on apt.postgresql.org with credativ’s help. This work will be handled by our colleague Christoph Berg.
The release of PostgreSQL® 12 is expected in the next weeks.
This week version 1.3 of our PostgreSQL® appliance Elephant Shed was released.
The highlight of the new version is support for Red Hat Enterprise Linux 7 and CentOS 7. As is already the case for Debian, the appliance heavily relies on the postgresql-common infrastructure which was previously ported to RPM.
The well-known PostgreSQL® RPM packages from yum.postgresql.org are integrated into the system via pg_createcluster and can be administrated from the Elephant Shed web interface.
All other Elephant Shed components like pgAdmin4, Grafana, Prometheus, pgbackrest, Cockpit or shellinabox work in the same way as in the Debian version of the appliance. Only the SELinux functionality has to be deactivated in order to run pgAdmin4 and shellinabox as their packages do not support this.
Besides the port to RPM the appliance infrastructure was updated. The Prometheus node-exporter is now available in version 0.16 in which many metric names were adjusted to the Prometheus naming scheme. The Grafana dashboard was updated accordingly. The Apache configuration was switched from authnz_pam to authnz_external as the former is not available on CentOS and stable functionality could not longer be guaranteed on Debian Buster.
The next items on the Elephant-Shed roadmap are the integration of the REST-API in order to control particular components, as well as multi-host support so that several Elephant-Shed instances can be controlled simultaneously.
An overhaul of the user interface is planned as well.
The updated packages are available for download at packages.credativ.com. If Elephant-Shed was installed already, the updates are provided via apt as usual.
The open-source PostgreSQL® appliance Elephant-Shed is developed by credativ and is increasingly popular, as the most important compontents for the administration of PostgreSQL® servers are already included. Adjustments and extensions can be done at any time.
Comprehensible technical support for Elephant-Shed is offered by credativ including guaranteed service-level agreements and optional 365 days and 24/7 hours.
Today PostgreSQL® version 11 was released. The new release brings improvements in many areas.
Since version 9.6 query plans can be executed on multiple CPU cores in parallel, this is now supported for other plan types, especially the creation of B-tree indexes. Sequential scans and UNION queries have also been improved.
Brand new is the possibility to optimize queries via Just-in-Time Compilation (JIT). When PostgreSQL® is compiled, the source code is stored as LLVM bit code. When executing a query whose planner cost exceeds a limit, libllvm then translates this bit code into native machine code specifically for that query. Since all used data types etc. are known in advance, the machine code eliminates all case distinctions that are generally necessary. The feature is disabled by default and can be enabled with “SET jit = on;”. In PostgreSQL® 12 it should then be active by default.
Until now only functions could be defined on SQL level. New are now procedures which can manage BEGIN/COMMIT independently. Batch operations can now be completely transferred to the database side.
Table Partitioning section has been further improved to support hash partitioning. Integration with partitions using postgres_fdw has also been improved. It is now possible to create a default partition that holds data that does not fall into any of the existing partitions.
This Amp goes to 11!
Other improvements include the ability to add columns with default values to tables without having to rewrite the table completely. “Covering Indexes” allows more index accesses than “Index Only Scan”. Window functions now also support the RANGE and GROUPS keywords.
More information can be found in the release announcement of the PostgreSQL® project.
The shirt is available for free at our booth. If you don’t attend the conference, you can order a shirt. The profits will be donated to the PostgreSQL® project.
The credativ PostgreSQL® Appliance is now available for download as a completely free Open Source solution. The new appliance called ‘Elephant Shed PostgreSQL® Appliance‘ offers a tremendous ease of use for enterprise PostgreSQL® operations. The long-term maintenance of the appliance will be handled by the PostgreSQL® experts of credativ. Prepared images for the major cloud and virtualization platforms are already in work and will be released soon for VMWare, Virtualbox, Vagrant and for the cloud platforms Microsoft Azure, Amazon Web Services and Google Cloud Platform.
Elephant Shed builds on proven components, which are published exclusively under recognized Open Source licenses. These tools are an effective support for the management of a PostgreSQL® server. All components used are pre-installed and integrated into the integral automation system. The majority of these tools can be used via a comfortable web interface.
Service and Support
For Elephant Shed, credativ offers comprehensive technical support with guaranteed service level agreements, which is also available 24 hours a day, 365 days a year. Support during installation and integration, as well as an introduction to Elephant Shed is of course also part of credativ’s services.
Please do not hesitate to contact us if you have any questions about Elephant Shed and our service and support services.
This article was originally written by Philip Haas.
“The addition of credativ brings another important Open Source pillar into the OpenChain community and boosts our mission to make Open Source Software compliance easier, more understandable and more transparent,” says Shane Coughlan, OpenChain Program Manager. “credativ is one of the early supporters of Free and Open Source Software in our community. They have engaged at a unique time. We are looking forward to working closely together to channel influence in the European, American and Indian market as we scale OpenChain self-certification.”
“It is an important step for us to join the OpenChain community,” says Dr Michael Meskes, CEO of credativ. “Through our extensive experience with Open Source in business, we work with a lot of different companies and know how to improve upon the positive impact that Open Source provides, not only in technical but also compliance issues. Therefore we would like to further extend the reputation of the OpenChain Project by expressing adherence to their specifications. We think that this leads to more and more companies gaining trust and confidence in Free and Open Source Software. One of the key aspects of the OpenChain certification is, that it builds trust, which, already rooted in the core principles of Free and Open Source Software, thusly provides an essential contribution to the general acceptance of Open Source.“
Thus the project builds trust in Free and Open Source Software by making Open Source license compliance simpler and more consistent.
The OpenChain Specification defines a core set of requirements every quality compliance program must satisfy. The OpenChain Curriculum provides the educational foundation for Open Source processes and solutions, whilst meeting a key requirement of the OpenChain Specification. OpenChain
Conformance allows organizations to display their adherence to these requirements.
The result is that Open Source license compliance becomes more predictable, understandable and efficient for participants of the software supply chain.
credativ is a independent consulting and services company operating in Germany, Spain, India, the Netherlands and the USA. Since the founding in 1999, credativ has been offering comprehensive services and technical support for the implementation and operation of Open Source Software in business applications. For more information, visit https://www.credativ.com
About The Linux Foundation:
The Linux Foundation is the organization of choice for the world’s top developers and companies to build ecosystems that accelerate open technology development and commercial adoption. Together with the worldwide open source community, it is solving the hardest technology problems by creating the largest shared technology investment in history. Founded in 2000, The Linux Foundation today provides tools, training and events to scale any open source project, which together deliver an economic impact not achievable by any one company. More information can be found at https://www.linuxfoundation.org.
The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our trademark usage page: https://www.linuxfoundation.org/trademark-usage. Linux is a registered trademark of Linus Torvalds.
OpenChain Blogpost: https://www.openchainproject.org/news/2017/08/07/openchain-welcomes-credativ
Debian 9 “Stretch”, the latest version of Debian, is about to be released and after a full-freeze on February 5th everyone is trying its best to fix the last bugs that are left.
Upon entering the final phase of development in February the test version was “frozen” so that no more packages could be added or removed without the approval of the release team.
However, Stretch has some bugs left, which need to be resolved until the release date, especially the so called release critical bugs (RC). For this purpose, numerous Debian developers host worldwide meet ups.
These meet ups are a long standing tradition and are lovingly called “Bug Squashing Party”. Despite the cute name, these events usually turn out to be one the most focused, intense and hard working days in the life cycle of a new Debian version. Pressured by the upcoming release date, everyone gets together to get rid of the nasty release critical bugs and focus on unfinished packages.
The Open Source Support Center employs the likely biggest number of European Debian developers in one place. Therefore credativ GmbH is providing the location and technical infrastructure for everyone who decided to join the Bug Squashing Party.
We hope that this year’s meeting is going to be as successful as in the previous years. Developers from all neighbouring countries took part in past events and some even found their future employer.
If you would like to participate, feel free to sign up!
We are looking forward to your visit.
Here is the announcement on the mailing-list:
Here is the entry in the Debian wiki:
This article was originally written by Philip Haas.