1. Why did Amsterdam started to work on procurement clauses and in which context?

Governments are increasingly using AI and algorithmic systems to provide public services. The City of Amsterdam is using algorithms in some primary city tasks, like for example parking control and acting on reporting issues in public space from citizens. Amsterdam has several instruments to guarantee the use of AI in a secure and responsible manner. We want to ensure that the digital human rights of Amsterdam’s citizens, entrepreneurs and visitors are safeguarded; not only offline but also online. This requires ethical principles like participation and inclusion during the design, development, application and operation of AI and algorithmic systems. Being transparent is an important condition for citizens trust in the government, research also supports this. As more and more public services start to depend on AI and algorithmic systems, and we want to be transparent about the decision-making process, transparency about AI is needed. The AI-clauses are setting standards, creating obligations and establishing responsibilities for trustworthy, transparent and accountable development and use of algorithms in services. In the last couple of years, many guidelines and frameworks have been published on algorithmic accountability. A notable example is the Ethics guidelines for trustworthy AI by the High-Level Expert Group on AI advising the European Commission but of course there are many other examples. All these frameworks have in common that they name transparency as a key principle of trustworthy AI and algorithmic systems. The next step was, how to operationalize this concept of transparency. The city has designed several instruments for trustworthy use of algorithms who are summed up in the ‘Algorithm Life Cycle’ policy. One of these instruments are the procurement conditions which is a joint force result of several Dutch and international experts published in November 2020. What is good to know is that at the time the first version of the Amsterdam Clauses was developed, there was no concrete legislation in the field of AI. There were many guidelines under development at that time, such as the The Ethics Guidelines for Trustworthy Artificial Intelligence (AI) of the High-Level Expert Group on Artificial Intelligence. Amsterdam wanted to go a step further and develop a product that could actually be used by European cities that want to set limits on the use of AI in practice.   After all, procurement terms are a relatively easy tool with which a lot of impact can be achieved. This is a role that buyers can play on this spectrum. They know the practice and are able to develop practical tools relatively quickly and deploy those tools as well.

2. What were the first experiences working on the clauses and what kind of feedback did the City receive making them?

 In developing the Amsterdam Clauses, we worked closely with a group of experts from various disciplines. In addition to the legal expertise of my office, AI experts from KPMG also contributed to the development of the Amsterdam Clauses. In the initial phase, much attention was paid to questions such as how to define algorithm, what forms of transparency can we distinguish, and how to deal with issues such as IP and data ownership.

 Various versions of the Amsterdam Clauses were discussed at a very early stage with a delegation of IT suppliers. They were remarkably positive about the Amsterdam Clauses. They understood very well that the use of algorithms in the performance of public tasks entails the need to provide a degree of transparency. However, they were critical of the degree of transparency that could be offered and asked that attention be paid to the protection of trade secrets.

3. Why did you contacted the European Commission for further collaboration on the topic? How did the collaboration start?

The ambition of this project has always been to show it is possible to operationalize general guidelines for AI ethics to ensure the digital human rights are safeguarded while making use of the advantages that digitalization may bring to a city. Because of interest from other players in Europe with regards to the Amsterdam practices, further promotion of ethical AI by having an European standard for public procurement of AI And algorithmic systems was a perfect next step. For this reason, we, DG CNECT and DG GROW of the EC jointly decided to announce this collaboration by holding a webinar. Here, Amsterdam shared it’s experiences on the AI-clauses and it’s conditions were in depth presented by Jeroen Naves. This was a first step in ultimately creating a draft version of the European AI Clauses.

4.  What were the most difficult issues to convert the “Amsterdam version” into a European draft? What was the role of the upcoming legal framework on AI?

In drafting the European version of the Amsterdam Clauses, we have tried to align as much as possible with the proposed AI Act. In this context, there was close cooperation with the legal experts of DG CNECT who are involved in the drafting of the AI Act.

The ultimate goal is to arrive at a set of conditions that will be useful for governments wishing to anticipate the AI Act's entry into force. In that context, it is important to realize that the AI Act largely contains best practices in the area of regulating AI. Aside from the fact that the AI Act will enter into force at some point, the proposed AI Act also contains general standards that can be applied in the procurement of AI.

In drafting the European variant, we encountered the following challenges:

- The proposed AI Act focuses particularly on high risk AI systems, and thus so does the latest version of AI Clauses. Therefore, the AI Clauses may in some respects set too heavy requirements for low risk AI systems. This can be solved by using some provisions of the AI Clauses and not others in the case of the purchase of a low-risk AI System. In the near future, I hope to be able to formulate more guidelines for this.

- The proposed AI Act is still under development. There will be new versions of the AI Act in the coming years. This means that the AI Clauses will also need to be updated all the time.

- The AI Clauses are written in such a way that they can be added to more generic purchasing conditions.

For example, the AI Clauses do not contain provisions on applicable law or liability. However, a general provision on IP is included.

- The AI Clauses do not take into account any national law rules regarding the use of AI.

5. What is your dream concerning the further development of the AI-clauses? What would be the best scenario for you?

Lydia: Commitment and implementation of the AI-clauses in the first place for two reasons:Firstly, to collect feedback from various stakeholders to be used in the next version of the European AI purchase conditions and feedback which can be shared by participating in the Community of Practice. Secondly, to promote the necessity of using trustworthy AI and algorithmic systems. Implementation of the clauses promotes the use of AI aligned with digital human rights, ethical standards, the public interest and simultaneously by improving people’s quality of life.

Jeroen:Ultimately, the goal is for the AI Clauses to become the standard worldwide in the area of AI procurement. This does require that the AI Clauses become more dynamic and flexible so that they can be used in all kinds of situations where AI is procured. In doing so, it is of course important that the AI Clauses are sufficiently applicable, also for buyers who do not procure AI every day.

Lydia-José (Lydia) Prinsen is an Innovation Manager at the Chief Technology Office of the Municipality of Amsterdam.

Jeroen Naves specialises in technology and the law. His activities include drawing up and assessing IT contracts, litigating IT disputes and advising on data exchange and innovation. Jeroen litigates before the civil courts in both proceedings on the merits and preliminary relief proceedings. He is experienced at resolving IT disputes and bringing complex IT procurement processes to a successful conclusion. Jeroen is a trusted advisor to businesses, government authorities and public institutions. Clients prize his hands-on mentality.