Thursday, June 4, 2015

Presenting at OGH DBA / Middleware day 2015

Deliver my presentation "Dynamically scale Weblogic in a private Cloud" on the OGH DBA / Middleware day 2015 today at 13:15.

Friday, April 10, 2015

Creating and scaling Dynamic Clusters using wlst

In my previous article, Creating and scaling Dynamic Clusters in Weblogic 12c, I described the creation and scaling of Dynamic Clusters. I used the Weblogic Console to create the Dynamic Clusters and change the number of servers.

Most of the time you will use some wlst scripting to create and manage your Weblogic environments.
In this article I will show you how to create Dynamic Clusters en how you can scale them.

The example scripts from the Oracle documentation where used as base for the following script.
It is just a simple create script to show you how easy it is to create a Dynamic Cluster via wlst. So no fancy functions and exception handling in there. Yet ...

createDynamicCluster.py
print '--- Set properties for dynamic Cluster creation'
clusterName='dyna-cluster'
serverTemplate='dyna-server-Template'
serverNamePrefix='dyna-server-'
listenAddress='192.168.100.4${id}'
listenPort=8000
listenPortSSL=9000
maxServerCount=2

print '--- Connect to the AdminServer'
try:
  connect('weblogic','Welcome01','t3://hostname.domain.local:7001')
except err:
  print "--- Can't connect to AdminServer, "+err
  sys.exit(2)

print '--- Start an edit session'
edit()
startEdit()

print '--- Creating the server template '+serverTemplate+' for the dynamic servers and set the attributes'
dynamicServerTemplate=cmo.createServerTemplate(serverTemplate)
dynamicServerTemplate.setListenAddress(listenAddress)
dynamicServerTemplate.setListenPort(listenPort)
dynamicServerTemplateSSL=dynamicServerTemplate.getSSL()
dynamicServerTemplateSSL.setListenPort(listenPortSSL)

print '--- Creating the dynamic cluster '+clusterName+', set the number of dynamic servers and designate the server template to it.'
dynamicCluster=cmo.createCluster(clusterName)
dynamicServers=dynamicCluster.getDynamicServers()
dynamicServers.setMaximumDynamicServerCount(maxServerCount)
dynamicServers.setServerTemplate(dynamicServerTemplate)

print '--- Designating the Cluster to the ServerTemplate'
dynamicServerTemplate.setCluster(dynamicCluster)

print '--- Set the servername prefix to '+serverNamePrefix
dynamicServers.setServerNamePrefix(serverNamePrefix)

print '--- Set Calculate Listen Port and Machinename based on server template'
dynamicServers.setCalculatedMachineNames(true)
dynamicServers.setCalculatedListenPorts(true)

print '--- Save and activate the changes'
save()
activate()
serverConfig()


Running the script with wlst will produce the following output and will create a Dynamic Cluster with two Dynamic Servers.

[oracle@wls01 ~]$ ${WL_HOME}/common/bin/wlst.sh createDynamicCluster.py
Initializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

--- Set properties for dynamic Cluster creation
--- Connect to the AdminServer
Connecting to t3://wls01.domain.local:7001 with userid weblogic ...
Successfully connected to Admin Server "AdminServer" that belongs to domain "demo_domain".

Warning: An insecure protocol was used to connect to the
server. To ensure on-the-wire security, the SSL port or
Admin port should be used instead.

Start an edit session
Location changed to edit tree. This is a writable tree with
DomainMBean as the root. To make changes you will need to start
an edit session via startEdit().

For more help, use help('edit')

Starting an edit session ...
Started edit session, please be sure to save and activate your
changes once you are done.
--- Creating the server template dyna-server-Template for the dynamic servers and set the attributes
--- Creating the dynamic cluster dyna-cluster, set the number of dynamic servers and designate the server template to it.
--- Designating the Cluster to the ServerTemplate
--- Set the servername prefix to dyna-server-
--- Set Calculate Listen Port and Machinename based on server template
--- Save and activate the changes
Saving all your changes ...
Saved all your changes successfully.
Activating all your changes, this may take a while ...
The edit lock associated with this edit session is released
once the activation is completed.
Activation completed

As you might expect, it is way faster than clicking through the Weblogic Console.


Next step will be to scale the Dynamic Cluster up to four Dynamic Servers.

scaleDynamicCluster.py
print '--- Set properties for dynamic Cluster creation'
clusterName='dyna-cluster'
maxServerCount=4

print '--- Connect to the AdminServer'
try:
  connect('weblogic','Welcome01','t3://wls01.domain.local:7001')
except err:
  print "Can't connect to AdminServer, "+err
  sys.exit(2)

print '--- Start an edit session'
edit()
startEdit()

print '--- Change the maximum number of dynamic servers'
cd('/Clusters/%s' % clusterName )
dynamicServers=cmo.getDynamicServers()
dynamicServers.setMaximumDynamicServerCount(maxServerCount)

print '--- Save and activate the changes'
save()
activate()
serverConfig()

Running the script with wlst will produce the following output and will scale up to four Dynamic Servers.
[oracle@wls01 ~]$ ${WL_HOME}/common/bin/wlst.sh scaleDynamicCluster.py

Initializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

--- Set properties for dynamic Cluster creation
--- Connect to the AdminServer
Connecting to t3://wls01.domain.local:7001 with userid weblogic ...
Successfully connected to Admin Server "AdminServer" that belongs to domain "demo_domain".

Warning: An insecure protocol was used to connect to the
server. To ensure on-the-wire security, the SSL port or
Admin port should be used instead.

--- Start an edit session
Location changed to edit tree. This is a writable tree with
DomainMBean as the root. To make changes you will need to start
an edit session via startEdit().

For more help, use help('edit')

Starting an edit session ...
Started edit session, please be sure to save and activate your
changes once you are done.
--- Change the maximum number of dynamic servers
--- Save and activate the changes
Saving all your changes ...
Saved all your changes successfully.
Activating all your changes, this may take a while ...
The edit lock associated with this edit session is released
once the activation is completed.
Activation completed

As mentioned before, the scripts are very limited and just show you how easy it is to create Dynamic Clusters using wlst. The scripts can be made as comprehensive as you need (want) them to be.
I will create some more examples and post them as I get them ready.

Imagine the possibilities when you create scripts you can connect to your monitoring system. Capacity on demand!

Tuesday, April 7, 2015

Creating and scaling Dynamic Clusters in Weblogic 12c

Introduced in Weblogic 12.1.2, dynamic clusters is a great feature to scale your private cloud.
Dynamic clusters provides you easy scaling of Weblogic clusters by adding and removing managed server instances on demand. They contain one or more dynamic servers. These dynamic servers are based on a single servertemplate that guarantees that every member of the cluster is exactly the same.

Creating Dynamic Clusters


Let's take a look at some of the possibilities as we create a dynamic cluster.

I have created a virtualbox environment.
This environment consists of four VM's with the following specs.

  • 2 vCPU's
  • 4 Gb memory
  • 50 Gb disk
  • Oracle Linux 6.6
  • Java 1.7.0_75
  • Weblogic 12.1.3.0.2

I created a simple domain called demo_domain with only an AdminServer and four machines.
After unpacking the domain to the four servers, the nodemanagers where started and are reachable by the AdminServer.


Now let go through the process of creating a dynamic cluster.

Open the Weblogic Console and navigate to Environment -> Clusters
Lock and Edit the domain in the Change Center
note. I make it a good practice always creating a domain in production mode, even in Development and Test.

Create a new dynamic cluster
New -> Dynamic Cluster

Provide the Clustername
Cluster name: dyna-cluster
Click Next

We will start of with a cluster containing two dynamic servers.
Number of Synamic Servers: 2
Server Name Prefix: dyna-server-
Click Next

For this demo all machines will take part.
Select 'Use any machine configured in this domain'
Click Next

Assign each dynamic server unique listen ports
Listen Port for First Server: 8000
SSL Listen Port for First Server: 9000
Click Next

Summary screen
Click Finish

With the creation of the Dynamic Cluster there is also a Server Template created for it.


Server templates

A single server template provides the basis for the creation of the dynamic servers. Using this single template provides the possibility of every member being created with exactly the same attributes. Where some of the server-specific attributes like Servername, listen-ports, machines, etc. can be calculated based upon tokens.
You can pre-create server templates and let Weblogic clone one when a Dynamic Cluster is created.
When none is available a server template is create with the Dynamic Cluster. The name and the listen ports are the only server template attributes that you provide during Dynamic Cluster creation.

Before we activate the changes to the domain, we are going to make a change to the server template.
As an example we are going to demonstrate the use of tokens for server-specific configuration. 

Navigate to Environment -> Clusters -> Server Templates

Click on the name: dyna-server-Template

We are going to use the ${ID} token in the Listen Address
Listen Address: 192.168.100.4${ID}
Click Save

The last digit of the listen address is used to make the listen address dynamic.

Activate changes in the Change Center of the Weblogic Console.
After activation the cluster and two managed servers are created.

We can now start the two servers.

In the previous steps we have added a dynamic cluster with two dynamic servers, based on a single server template, to the domain.

Scaling a Dynamic Cluster


When the capacity is insufficient and you need to scale-up, you can add dynamic servers on demand.
It requires only a few clicks.

Navigate to Environment -> Clusters
Click dyna-cluster

On the Configuration tab go to the Servers tab
Change the Maximum Number of DynamicServers to: 4
Click save

Activate changes in the Change Center of the Weblogic Console.
After activation two Dynamic Servers are added to the Dynamic Cluster.

Start the two new Dynamic Servers and you have doubled you capacity.



Scaling down works exactly the same.
Just lower the Maximum Number of DynamicServers and activate.

A few points to keep in mind when scaling up or down.

Up
  • New dynamic servers are not started upon creation
  • Think before you act with the use of Tokens.
    For example.
    In our demo, the number of Dynamic servers can't grow beyond nine servers, since we use the ${ID} as last digit of the listen address.

Down
  • Dynamic Servers above the new Maximum have to be shutdown before the change can be activated.
  • Dynamic Servers are removed in order, Last -> First
    (In our demo dyna-server-4 gets removed first, then dyna-server-3, etc..)
  • You cannot remove a Dynamic Server directly from the Environment -> Servers page



Tuesday, January 13, 2015

Upgrade OVM 3.2 to 3.3

Lately several people asked me how they should go about with upgrading their OVM 3.2.x environment to 3.3.x
They have the backend repository running on an Oracle Database, which is no longer supported. In release 3.3, only MySQL is supported as a backend repository database.
They were wondering how the migration process will take place.
Furthermore the documentation states that there is only one way to upgrade the OVM Server from 3.2 to 3.3.

See Chapter 5.1 of the Installation and Upgrade Guide for Release 3.3

If you are upgrading from Oracle VM Server Release 3.2 or earlier to Release 3.3:

You must set up two Yum repositories, one as a transitional repository containing the packages for Release 3.2, and a second containing the target packages for Release 3.3. When the Yum repositories are in place, you can run the UpgradeServers.py script on the Oracle VM Manager host to batch update servers as required

This would suggest that the ‘old fashion’ method of booting the OVM Server from CD no longer works.
Well I have tested the upgrade the ‘old fashion’ way and I can say that it will work with a few tweaks.
However, chapter 5.4.4 states that this method is not supported. So let’s see how the supported upgrade methods are carried out.

Test Environment

Let’s start with the setup of a small test environment based on VirtualBox VM’s.

Virtualbox Machines
Hostname eth0 (LAN) eth1 (Storage)
storage1.domain.local 10.10.10.170
ovm01.domain.local 192.168.100.17110.10.10.171
ovmmanager.domain.local192.168.100.170






ISCSI Storage on storage1.domain.local (simulated by Openfiler)
Volumes (LUN’S)SizeDescription
poolfs 20GbOVM Pool filesystem
repfs 50GbStorage Repository





OVM Server
OVM Server 3.2.8 installed on ovm01.domain.local
4 Gb memory
12 Gb disk
1 network interface for LAN connection
1 network interface for Storage connection

1 Virtual guest running OEL 6.5

OVM Manager
OVM Manager 3.2.8 installed on ovmmanager.domain.local
4 vCPU’s
8 Gb memory
25 Gb disk
1 network interface for LAN connection

Oracle 11.0.2.4 database as backend repository
Instance name: ovsdb

Upgrade OVM Manager


As with all releases, the upgrade starts with upgrading the OVM Manager.
The upgrade of OVM Manager is pretty straightforward.

It involves

  • Upgrading the OVM Manager software from 3.2.8 to 3.3.1
  • Migrating ovsdb from Oracle database to MySQL database

Make sure there is enough diskspace available for the upgrade. The runInstaller states that 7100MB should be available in the filesystem where the current OVM Manager is installed.

Start the installer from the mounted iso or from the cd.
In the example below I have mounted the iso on /media

[root@ovmmanager media]# ./runInstaller.sh

Oracle VM Manager Release 3.3.1 Installer

Oracle VM Manager Installer log file:
/var/log/ovmm/ovm-manager-3-install-2014-10-23-214811.log

Please select an installation type:
1: Install
2: Upgrade
3: Uninstall
4: Help

   Select Number (1-4): 2

Verifying upgrading prerequisites ...
Starting Upgrade ...

Reading database parameters from config ...

Choose option 2: Upgrade

Note. You can also start the installer with the Upgrade option directly on the commandline
./runInstaller.sh --installtype Upgrade

Provide the database password of the current ovs schema in the Oracle database.
Database Repository
==========================
Please enter the current Oracle VM Manager database password for user ovs: *******

Provide the current admin password, this is the password you use to logon to the OVM Console.
Oracle VM Manager application
=============================
Please enter the current Oracle VM Manager application password for user admin: ********

Now provide a password for the weblogic user for the new Weblogic 12c install. You can choose to make up a new password or just use the same as the old installation.
Oracle Weblogic Server 12c
==========================
Please enter a *new* Oracle WebLogic Server 12c user password for user weblogic: ********
Please enter a *new* Oracle WebLogic Server 12c user password for user weblogic (confirm): ********

Provide a FQDN name for which a SSL certificate will be created.
Please enter your fully qualified domain name, e.g. ovs123.us.oracle.com, (or IP address)
of your management server for SSL certification generation, more than one IP address are
detected: 192.168.100.170 10.0.3.15 [ovmmanager.domain.local]: ovmmanager.domain.local

The installer verifies the configuration and offers you the choice to continue with the upgrade or abort the installation.
Verifying configuration ...

Upgrading from version 3.2.8.733 to version 3.3.1.1065

Start upgrading Oracle VM Manager:
   1: Continue
   2: Abort

   Select Number (1-2): 1

The upgrade will now proceed.
The console output scrolls down on your screen. I will not bother you with the full output.
You can monitor the progress in another terminal session via:
tail -f /var/log/ovmm/ovm-manager-3-install-YYYY-MM-DD-HHMISS.log

At the end of the upgrade an installation summary is presented, showing the relevant informantion.
Installation Summary
--------------------
Database configuration:
  Database type               : MySQL
  Database host name          : localhost
  Database name               : ovs
  Database listener port      : 49500
  Database user               : ovs

Weblogic Server configuration:
  Administration username     : weblogic

Oracle VM Manager configuration:
  Username                    : admin
  Core management port        : 54321<
  UUID                        : 0004fb00000100002482670af80b9a35

Passwords:
There are no default passwords for any users. The passwords to use for Oracle VM Manager, Database, and Oracle WebLogic Server have been set by you during this installation. In the case of a default install, all passwords are the same.

Oracle VM Manager UI:
  https://ovmmanager.domain.local:7002/ovm/console
Log in with the user 'admin', and the password you set during the installation.

Note that you must install the latest ovmcore-console package for your Oracle Linux distribution to gain VNC and serial console access to your Virtual Machines (VMs).
Please refer to the documentation for more information about this package.

For more information about Oracle Virtualization, please visit:
  http://www.oracle.com/virtualization/

Oracle VM Manager upgrade complete.

Please remove configuration file /tmp/ovm_configX4zw9Q.

A few points to mention that answer some customer questions.

During the upgrade the content of the Oracle database is automatically exported.
Exporting core database, please be patient this may take a long time  ...

Further down the upgrade track, after MySQL database has been created, the export is imported into it.
Importing core database, please be patient this may take a long time ...

Don’t forget to shutdown and delete the Oracle database after the successful upgrade.

After the upgrade to 3.3 the OVM Manager is set to communicate with OVM Server using TLS. Since this was not the case with to OVM Server 3.2, we disable this option until all OVM Servers are upgraded.
To communicate with OVM Servers 3.2, disable TLS1 for SSL-based authentication in release 3.3.
vi /etc/sysconfig/ovmm
USETLS1=NO
After all OVM Servers have been upgraded, change it back to
USETLS1=YES
Remove any temporary files left by the installer.
rm -Rf /tmp/workdir_sel*
rm -Rf /tmp/ovm*

We can now proceed with the upgrade of the OVM Servers.

Upgrade OVM Server


Before you begin.

Make sure you have downloaded the latest installation media from edelivery.
According to an unpublished bug, there is a defect on initial releases of Oracle VM 3.3.1 installation media. This could make some servers hang on boot.

As mentioned earlier, the upgrade of OVM Server from release 3.2 or earlier to release 3.3 via booting from the CD is no longer supported.

We are going to create two Yum repositories.
A transitional repository containing 3.2 packages and a target repository containing 3.3 packages.
Both repositories will be created from the 3.3 installation media.

The repositories need to be accessible through http/https on any webserver.
If you don’t have a webserver available, you can use the Python SimpleHTTPServer module.
We will be using this module on the OVM Manager server as an example.

Creating the Yum repositories

Logon as root on the server.

Create a directory for the repository files.
mkdir -p /var/www/repos
Mount the iso on /media
mount -o loop OVS-3.3.1.iso /media
Copy the entire iso to the http root directory
cp -rp /media/* /var/www/repos/

Start the SimpleHTTPServer
cd /var/www
python –m SimpleHTTPServer 80
Serving HTTP on 0.0.0.0 port 80
Keep this session running until the upgrade has finished.

If you have any additional packages installed, see “5.4.1.2 Adding Packages to a Yum Repository” on how to add them to the repository.

Now configure the repositories in OVM Manager console.

Open the OVM Manager webconsole in a browser.

Go to : Tools and Resources -> Server Update Groups -> GlobalX86ServerUpdateConfiguration



Create the two repositories Transitional and Target
Click on the + to enter the “Create New Server Update Repository” screen.

Transitional repository

Name : 3.3_trans_repo
Repository name : 3.3_trans_repo
URL : http://ovmmanager.domain.local/repos/Transition
Enabled : Yes
Package Signature Type :GPG
Package Signature Key : http://ovmmanager.domain.local/repos/RPM-GPG-KEY-oracle-ol5









The Name must be exactly as above entered as the UpgradeServers.py will use this Name.

Target repository

Name : 3.3_ovs_repo
Repository name : 3.3_ovs_repo
URL : http://ovmmanager.domain.local/repos/Server
Enabled : Yes
Package Signature Type :GPG
Package Signature Key : http://ovmmanager.domain.local/repos/RPM-GPG-KEY-oracle









The Name must be exactly as above entered as the UpgradeServers.py will use this Name.

Upgrade the OVM Servers


After the repositories have been created in OVM Manager we can proceed with the upgrade of the OVM Servers.
The upgrade is performed by using the UpgradeServers.py script.
/u01/app/oracle/ovm-manager-3/ovm_tools/bin/UpgradeServers.py
For the upgrade from OVM Server 3.2 to 3.3 this script is the required method.
It is possible to upgrade between 3.3.x versions using the webconsole, however the preferred method is using the script.

With the UpgradeServers.py script you can upgrade one or multiple OVM Servers or complete Server Pools at once.
See 5.4.2 Upgrading Oracle VM Server using the UpgradeServers.py Script for the syntax.

In our example I will upgrade the one OVM Server in the test setup.

On the OVM Manager server

cd /u01/app/oracle/ovm-manager-3/ovm_tools/bin/
./UpgradeServers.py -u admin -v ovm01.domain.local

Enter your OVM Manager password: ******
Tue Jan 13 10:52:03 2015 INFO:
Tue Jan 13 10:52:03 2015 INFO:  UpgradeServers script starting...
Tue Jan 13 10:52:03 2015 INFO:  OVM Manager version: 3.3.1.1065
Tue Jan 13 10:52:03 2015 INFO:  Command line args: ['UpgradeServers.py', '-v', 'ovm01.domain.local']
Tue Jan 13 10:52:04 2015 INFO:  Server: ovm01.domain.local. Transition Server Update Repository: 3.3_trans_repo,        URL/path: http://ovmmanager.domain.local/repos/Transition
Tue Jan 13 10:52:04 2015 INFO:  Server: ovm01.domain.local. OVS Server Update Repository: 3.3_ovs_repo, URL/path:        http://ovmmanager.domain.local/repos/Server
Tue Jan 13 10:52:04 2015 INFO:  Getting update packages list in Server Update Repository: 3.3_ovs_repo, using old       est server: ovm01.domain.local, version: 3.2.8-736
Tue Jan 13 10:52:07 2015 INFO:  Checking servers for non-native packages (those installed after initial server in       stallation)
Tue Jan 13 10:52:08 2015 INFO:  Non-native package status:
Tue Jan 13 10:52:08 2015 INFO:  Non-native package              Status in 3.3_ovs_repo
Tue Jan 13 10:52:08 2015 INFO:  -----------------------------   ----------------------
Tue Jan 13 10:52:08 2015 INFO:  openscap                        OK: package exists
Tue Jan 13 10:52:08 2015 INFO:  openscap-python                 OK: package exists
Tue Jan 13 10:52:08 2015 INFO:  openscap-utils                  OK: package exists
Tue Jan 13 10:52:09 2015 INFO:  No non-generic plug-ins have been found that are in use on existing storage arrays
Type YES to continue with upgrade: YES

Tue Jan 13 10:54:00 2015 INFO:  Evaluating server: ovm01.domain.local, version: 3.2.8-736, for upgrading. [1 of 1 servers].
Tue Jan 13 10:54:00 2015 INFO:  Disabling Server Update Repository: 3.3_ovs_repo
Tue Jan 13 10:54:00 2015 INFO:  Enabling Server Update Repository: 3.3_trans_repo
Tue Jan 13 10:54:00 2015 INFO:  Waiting up to 25 seconds for updates of the Server Update Repositories to complete on server ovm01.domain.local.
Tue Jan 13 10:54:02 2015 INFO:  Finished updating Server Update Repositories.
Tue Jan 13 10:54:03 2015 INFO:  Starting upgrade of server: ovm01.domain.local, type: X86_64, version: 3.2.8-736, using Server Update Repository: 3.3_trans_repo
Tue Jan 13 10:54:04 2015 INFO:  No VMs are on server: ovm01.domain.local, starting server upgrade.
Tue Jan 13 10:55:14 2015 INFO:  Waiting for upgrade of server: ovm01.domain.local, to complete. The server is performing the upgrade. [STOPPING]
Tue Jan 13 10:55:44 2015 INFO:
……
Tue Jan 13 11:02:46 2015 INFO:  Waiting for upgrade of server: ovm01.domain.local, to complete. The server is performing the upgrade. [STOPPING]
Tue Jan 13 11:03:16 2015 INFO:  Waiting for upgrade of server: ovm01.domain.local, to complete. The server is rebooting after the upgrade. [STOPPED]
Tue Jan 13 11:03:46 2015 INFO:  Waiting for upgrade of server: ovm01.domain.local, to complete. The server is rebooting after the upgrade. [STOPPED]
Tue Jan 13 11:04:16 2015 INFO:  Server: ovm01.domain.local, upgraded successfully to version: 3.3.1-1065 (using Server Update Repository: 3.3_ovs_repo).
Tue Jan 13 11:04:16 2015 INFO:  Evaluating if any storage arrays need their plug-ins updated.
Tue Jan 13 11:04:16 2015 INFO:  No plug-ins were found that needed updating.
Tue Jan 13 11:04:16 2015 INFO:  Log file is available at /u01/app/oracle/ovm-manager-3/ovm_tools/bin/UpgradeServers.log
Tue Jan 13 11:04:16 2015 INFO:  UpgradeServers script stopping...
The OVM Server has been upgraded to 3.3.1



Related Oracle Documention:
Oracle VM Installation and Upgrade Guide for Release 3.3

Related Oracle support notes:
Alert: Verify Sufficient Space in /boot on Your Hypervisor/dom0 Before Upgrading to Oracle VM 3.3.x (Doc ID 1927958.1
During/After the Upgrade to Oracle VM 3.3.1 Using a Dell optiplex 755/780 Machine Does Not Boot Anymore (Doc ID 1932065.1)
Oracle VM Server 3.3.1 Installation Hangs (Doc ID 1918017.1)